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I. 


INTRODUCTION 


A.  BACKGROUND 

1.  Deficient  Contractor  Business  Systems 

In  the  world  of  Department  of  Defense  (DoD)  acquisition  and  contracting,  managing 
costs  is  critical  to  sustaining  program  success  and  preventing  misused  resources  in 
government  contracts.  According  to  the  Commission  on  Wartime  Contracting  (2009b), 
“[contractor  business  systems  and  internal  controls  are  the  first  line  of  defense  against  waste, 
fraud,  and  abuse”  (p.  1)  and  are  essential  to  managing  cost  risk  to  the  government.  In  2009, 
the  Commission  on  Wartime  Contracting  found  multiple  deficiencies  in  contractor  business 
systems  and  internal  controls.  As  a  result,  Section  893  of  the  National  Defense  Authorization 
Act  (NDAA)  for  Fiscal  Year  2011  mandated  the  Secretary  of  Defense  to  implement  a 
program  to  improve  the  oversight  of  contractor  business  systems. 

The  Defense  Federal  Acquisition  Regulation  Supplement  (DFARS;  2013)  Clause 
252.242-7005  identifies  six  contractor  business  systems:  accounting,  estimating,  purchasing, 
earned  value  management  (EVM),  material  management,  and  property  management.  In  the 
event  that  any  business  system  demonstrates  a  significant  deficiency  in  which  it  is  unable  to 
produce  reliable  cost  management  data,  demonstrating  risk  to  the  government,  contractor 
payments  of  up  to  5%  can  be  withheld  for  each  deficient  system  (DFARS,  2013).  Although 
the  DFARS  business  system  rule  identifies  six  systems,  this  paper  focuses  solely  on  the 
Earned  Value  Management  System  (EVMS). 

The  next  section  introduces  the  role  of  EVMS  within  the  Defense  Contract 
Management  Agency  (DCMA)  and  its  definition  of  significant  deficiency. 

2.  Earned  Value  Management  System 

EVM  is  the  DoD’s  primary  tool  for  measuring  acquisition  program  cost,  schedule, 
and  performance.  EVM  data  are  measured  against  the  contract’s  performance  measurement 
baseline  to  monitor  progress  and  ensure  the  least  cost  risk  to  the  government  (Snider  & 
Dillard,  2008).  Mahoney  and  Rego  (2011)  described  EVM  as  “a  systematic  approach  to  the 
integration  and  [measurement]  of  cost,  schedule,  and  performance  on  projects,  [which] 
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provides  an  early  warning  system  for  potential  threats  and  opportunities”  (p.  67). 
Correspondingly,  the  EVMS  validation  done  by  the  DCMA  serves  as  an  assessment  of  EVM 
data  to  certify  that  processes  are  in  accordance  with  the  32  EVMS  guidelines  as  issued  by 
American  National  Standards  Institute/Electronic  Industries  Association  (ANSI/EIA)  748 
(DCMA,  2011).  Dibert  and  Velez  (2006)  stated  that  the  guidelines  provide  a  practical 
approach  to  effective  project  management. 

Understanding  EVMS  compliance  and  deficiencies  first  requires  a  working 
knowledge  of  how  the  process  is  monitored,  annotated,  and  corrected.  The  next  section 
discusses  the  DCMA’s  EVMS  role,  EVMS  surveillance  procedures,  and  the  consequences  of 
noncompliance. 


a.  Defense  Contract  Management  Agency  EVMS  Surveillance 

The  DCMA  functions  as  the  DoD  executive  agent  and  compliance  authority 
for  EVMS  compliance  (DCMA,  2012a).  The  DCMA  conducts  EVMS  surveillance  upon 
contract  awards  for  all  contracts  valued  over  $20  million,  which  contain  the  EVMS  Federal 
Acquisition  Regulation  (FAR)  or  DFARS  clause  (DCMA,  2012a).  EVMS  surveillance 
ensures  that  contractor  EVM  data  are  in  compliance  with  the  32  EVMS  guidelines  as  issued 
by  ANSI/EIA  748.  However,  the  present  research  project  focuses  only  on  13  EVMS 
guidelines  that  senior  DCMA  EVM  specialists  have  identified  as  high-risk  guidelines  that 
require  annual  evaluations,  as  shown  in  Table  1. 
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Table  1 .  DCMA  13  High-Risk  EVMS  Criteria 
(Dibert  &  Velez,  2006) 


EVMS  Criteria 

13  “Hieh-Risk”  Criteria 

EVMS  Group  #  1:  Organization 

#  1 

Define  Authorized  Work  (WBS  Elements) 

#3 

Integrate  the  System 

EVMS  Group  #2:1 

banning,  Scheduling,  and  Budgeting 

#6 

Schedule  the  Work 

#7 

Identify  Products,  Milestones,  and  Indicators 

#8 

Plan  the  Performance  Measurement  Baseline  (PMB) 

#9 

Establish  Budgets  for  Work 

#  10 

Identify  Work  Packages 

#  12 

Identify  and  Control  Level  of  Effort 

EVMS  Group  #  4:  Analysis 

#23 

Analyze  Schedule  and  Cost  Variances 

#26 

Implement  Managerial  Actions 

#27 

Develop  Revised  Estimates  of  Cost  at  Completion 

EVMS  Group  #5:  Revisions 

#28 

Incorporate  Changes  into  Plans,  Budgets,  and  Schedules 

#32 

Document  Changes  to  the  PMB 

b.  Significant  Deficiencies 

The  DCMA’s  annual  surveillance  of  the  13  high-risk  EVMS  guidelines  is  the 
process  that  can  approve  or  disapprove  a  contractor’s  EVMS  (DCMA,  2012a).  Failure  to 
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meet  ANSI/EIA  748’s  standards  for  any  of  the  13  guidelines  results  in  a  significant 
deficiency  and  disapproval  of  the  EVMS.  DFARS  Clause  252.234-7002(a)  defines  a 
significant  deficiency  as  a  “shortcoming  in  the  system  that  materially  affects  the  ability  of 
officials  of  the  Department  of  Defense  to  rely  upon  infonnation  produced  by  the  system  that 
is  needed  for  management  purposes”  (DFARS,  2013,  §  252.242-7005). 

Therefore,  if  a  contractor’s  EVMS  fails  to  meet  one  or  more  high-risk 
guidelines  (1,  3,  6,  7,  8,  9,  10,  12,  23,  26,  27,  28,  and  32),  the  system  is  considered  to  have  a 
significant  deficiency  and  withholding  of  progress  payments  is  required  (DFARS,  2013). 
However,  the  DCMA’s  administrative  contracting  officers  (ACOs)  have  the  discretion  to 
determine  the  level  of  the  corrective  action  request  (CAR)  based  on  the  severity  of  the 
deficiencies  (CAR,  2013).  Table  2  describes  the  different  CAR  levels  and  action  required. 

Table  2.  CAR  Levels 
(CAR,  2013) 


CAR  Level  I 

Deficiencies  are  minor  in  nature  and  can 
be  quickly  corrected 

CAR  Level  II 

Deficiencies  require  time  to  correct  and  a 
corrective  action  plan  by  contractor  is 
required 

As  of  August  16,  2011,  the  new  business  system’s  rule  authorizes  DCMA 


ACOs  to  withhold  a  maximum  of  5%  of  contractor  progress  payments  on  all  DCMA  Level 
III  and  above  CARs  until  the  significant  deficiencies  have  been  corrected  (DFARS,  2013). 
Once  the  contractor  makes  progress  towards  implementing  their  corrective  action  plan,  the 
ACOs  have  the  discretion  to  reduce  the  withhold  percentage. 

Lockheed  Martin  is  the  first  contractor  from  which  the  DCMA  has  withheld 
payments  since  implementing  the  new  business  rule  (Fast,  2012).  In  October  2012,  the 
DCMA  withheld  $46.5  million  from  Lockheed  Martin  for  two  F-35  contracts  that  were 
delayed  in  schedule  and  overrunning  by  approximately  70%  in  cost  (Capaccio,  2012).  As  of 
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April  2013,  the  withhold  amount  increased  to  $130  million  as  a  result  of  failing  to  implement 
a  corrective  action  plan  for  the  deficiencies  (Capaccio,  2013).  The  progress  payments 
totaling  $130  million  equate  to  5%  of  funds  spent  by  Lockheed  Martin  for  both  contracts 
(Capaccio,  2013).  The  DCMA  has  no  plans  to  release  the  funds  until  progress  towards 
improving  significant  deficiencies  are  demonstrated  and  in  compliance  with  the  EVMS 
standard  guidelines. 

This  problematic  situation  with  Lockheed  Martin  cautions  DCMA  officials  to 
ensure  that  EVMS  compliance  and  significant  deficiencies  are  correctly  measured  in  order  to 
implement  the  business  system  payment  withholding  rule.  As  Naval  Postgraduate  School 
Senior  Lecturer  William  Fast  (2012)  stated,  “[T]he  financial  impact  and  materiality  of  the 
deficiency  is  difficult  to  quantify-since  it  is  the  inaccurate  and  unreliable  data  produced  by 
the  business  system-that  is  (in  fact)  the  deficiency”  (p.  17).  Therefore,  exploring  objective 
and  quantitative  ways  to  identify  a  significant  deficiency  and  calculating  government  risk  in 
dollars  is  the  reason  behind  this  study.  The  next  section  describes  the  motives  for  our 
research. 


B.  PURPOSE  OF  STUDY 

The  purpose  of  this  research  is  to  objectively  and  quantitatively  portray  EVMS  risk  in 
a  way  that  supports  a  monetary  withhold  decision  and  can  withstand  objection  (to  include 
litigation)  from  the  defense  contractor.  In  this  project,  we  hypothesize  that  using  quantitative 
risk  models  such  as  Value  at  Risk  (VaR)  methods  and  simplification  of  that  business  model 
for  use  by  contracting  officers  provides  value  to  DCMA  professionals  in  the  implementation 
of  the  new  business  rule.  The  project  seeks  to  provide  the  DCMA  with  a  more  defensible 
risk  value  model  as  the  basis  for  withholding  contractor  payments.  The  next  section 
describes  the  focus  and  priority  of  the  research  project. 

C.  RESEARCH  OBJECTIVES 

Three  principal  objectives  define  the  project  and  set  the  priorities  of  the  research: 

1.  determine  whether  the  13  EVMS  high-risk  guidelines  can  be  grouped  with 
respect  to  root  causes  (causality  of  risk), 
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•  evaluate  the  rank  or  natural  order  to  the  potential  severity  of  the 
deficiency  posed  by  these  guidelines,  and 

•  assess  the  degree  of  interdependence  or  causality  across  the  13  critical 
EVMS  guidelines. 

2.  Evaluate  which  quantitative  method(s)  can  be  used  to  calculate  risk  value  with 
respect  to  noncompliance  with  both  critical  and  noncritical  guidelines. 

•  Evaluate  which  quantitative  definition  of  significant  deficiency  is 
applicable. 

3.  Determine  the  relationship  of  risk  value  calculations  and  findings  of  EVMS 
noncompliance  with  the  (a)  probability  of  error,  (b)  magnitude  of  errors,  and 
(c)  adverse  impact  of  errors. 

•  Develop  a  deterministic  rule  set  that  yields  a  consistent  and  repeatable 
finding  of  significant  deficiency. 

The  next  section  suggests  how  relevant  stakeholders  can  benefit  from  the  research. 

D.  RESEARCH  BENEFITS 

Preventing  and  mitigating  fraud,  waste,  and  abuse  of  taxpayer  dollars  is  the  ultimate 
benefit  of  this  project.  In  2009,  the  Commission  on  Wartime  Contracting  found  that  the 
DCMA  lacked  an  effective  system  that  enforced  EVMS  compliance  among  contractors 
(DCMA,  2009b).  With  the  recent  federal  business  rule  in  effect,  DCMA  ACOs  now  have  the 
withholding  authority  to  motivate  contractors  to  comply;  however,  the  DCMA  is  under¬ 
resourced  to  implement  the  rule  objectively  (Fast,  2012).  Therefore,  a  quantitative  risk 
model  that  is  simplified  for  use  by  DCMA  contracting  officers  in  the  implementation  of  the 
recent  business  system  rule  can  provide  the  following  advantages: 

Objective  and  quantitative  assessment  of  the  13  critical  EVMS  guidelines 
justifies  appropriate  withhold  amounts  if  necessary,  thus  providing  a  litigation 
safety  net  for  the  DCMA. 

The  DCMA  has  a  simplified  risk  analysis  tool,  which  helps  alleviate  the  issue 
of  being  under-resourced  to  motivate  contractors  to  comply  with  EVMS 
surveillance  processes  and  criteria. 

A  risk  analysis  model  used  in  assessing  compliance  provides  informative  audit 
reports  that  help  ACOs  make  effective  contracting  decisions  well  beyond  the 
decision  to  withhold  payments. 
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•  Understanding  root  causes  and  any  correlations  between  the  critical  criterion 
deficiencies  can  provide  decision-makers  with  credible  data  to  improve 
current  EVMS  processes  and  procedures. 

E.  SUMMARY 

The  introduction  included  three  main  topics  of  discussion.  The  first  topic  we 
addressed  was  deficient  contractor  business  systems  and  the  new  business  rule  of 
withholding  5%  of  the  contractor  progress  payment  when  a  significant  deficiency  is  present. 
The  second  topic  focused  on  EVM  business  systems  and  the  DCMA’s  role  in  monitoring  a 
contractor’s  EVMS  for  compliance.  The  final  topic  discussed  the  purpose  of  our  research  and 
the  benefits  we  can  provide  to  the  DCMA.  The  next  chapter  establishes  a  working 
knowledge  by  exploring  EVMS  criteria  in  detail. 
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II.  DEFENSE  CONTRACT  MANAGEMENT  AGENCY’S  13  HIGH-RISK 
EARNED  VALUE  MANAGEMENT  SYSTEM  CRITERIA 

A.  PREFACE 

This  chapter  provides  a  detailed  description  of  the  selected  13  high-risk  guidelines  (1, 
3,  6,  7,  8,  9,  10,  12,  23,  26,  27,  28,  and  32)  that  the  DCMA  evaluates  for  significant 
deficiencies.  Understanding  the  criteria  in  better  detail  assists  in  determining  root  causes  of 
significant  deficiencies,  which  then  paves  the  way  to  developing  a  methodology  that 
identifies  the  ranking  of  severity  for  each  of  the  high-risk  noncompliant  guidelines. 

Each  essential  criterion  represents  logical  “best  practices”  in  both  the  DoD  and  the 
private  sector,  which  would  benefit  any  project  or  program  manager  (Fleming  &  Koppelman, 
2012).  The  13  guidelines  are  grouped  into  five  categories:  (1)  Organization,  (2)  Planning, 
Scheduling,  and  Budgeting,  (3)  Analysis,  (4)  Accounting,  and  (5)  Revisions.  The 
Accounting  category  is  excluded  from  the  13  high-risk  guidelines  because  it  is  a  separate 
business  system  under  the  surveillance  responsibility  of  the  Defense  Contracts  Audit  Agency 
(DCAA).  Therefore,  the  13  criteria  fall  under  only  four  of  the  five  categories. 

B.  EARNED  VALUE  MANAGEMENT  SYSTEM  HIGH-RISK 
CRITERIA 

1.  Group  1 — Organization 

The  first  category  of  Organization  contains  Criteria  1  and  3,  which  clearly  define  the 
range  of  requirements  prior  to  the  project  commencing  (Fleming  &  Koppelman,  2012). 
Fleming  and  Koppelman  (2012)  defined  the  criteria  as  follows: 


a.  EVMS  Criterion  1:  “Define  the  authorized  work  elements  for  the 
program.  A  work  breakdown  structure  (WBS),  tailored  for  effective 
internal  management  control,  is  commonly  used  in  this  process  (p. 
3698)” 

b.  Criterion  1  suggests  the  use  of  a  WBS  to  group  segments  of  work  in 
an  organized  manner  by  hierarchy.  All  deliverables  must  be 
documented  and  grouped  under  a  major  segment  in  the  WBS,  which 
ensures  that  project  managers  understand  their  scope  of 
responsibilities  in  order  to  measure  performance  (Fleming  & 
Koppelman,  2012).  A  common  issue  that  causes  a  significant 
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deficiency  in  this  criterion  is  a  poor  definition  of  the  project’s  range 
of  deliverables  (i.e.,  its  WBS,  product  specifications),  resulting  in 
wasted  resources  in  regard  to  time,  schedule,  and  performance 
(Fleming  &  Koppelman,  2012).  EVMS  Criterion  3:  “ Provide  for  the 
integration  of  the  company’s  planning,  scheduling,  budgeting,  work 
authorization,  and  cost  accumulation  processes  with  each  other,  and, 
as  appropriate,  the  program  WBS  and  the  program  organizational 
structure  (p.  3720-3728)” 

Criterion  3  requires  the  incorporation  of  organizational  management  in 
conjunction  with  the  WBS,  to  include  identifying  the  functional  teams  responsible  for  each 
work  package  (Fleming  &  Koppelman,  2012).  Additionally,  Criterion  3  requires  an 
information  database  process  to  be  created  for  all  functional  teams  within  the  project. 
Interestingly,  large  defense  contractors  typically  fail  at  satisfying  this  criterion  due  to 
functional  sections  maintaining  opposing  project  goals  and  processes  (Fleming  &  Koppelman, 
2012). 


2.  Group  2 — Planning,  Scheduling,  and  Budgeting 

The  second  category  of  Planning,  Scheduling,  and  Budgeting  contains  Criteria  6,  7,  8, 
9,  10,  and  12,  which  require  a  management  control  system  that  links  the  formal  planning, 
scheduling,  and  budgeting  of  a  project  into  a  PMB.  This  group  of  criteria  establishes  a 
project  baseline  that  allows  for  a  formal  means  of  project  discipline  and  assessment.  Fleming 
and  Koppelman  (2012)  defined  Criterion  6  as  follows: 
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a.  EVMS  Criterion  6:  “ Schedule  the  authorized  work  in  a  manner  that 
describes  the  sequence  of  work  and  identifies  the  significant  task 
interdependencies  required  to  meet  the  requirements  of  the  program 
(p.  3781-3788 )” 

b.  Criterion  6  requires  a  project  to  establish  a  project  master  schedule 
(PMS)  for  all  projects  within  the  WBS  (Fleming  &  Koppelman, 
2012).  The  PMS  must  reflect  key  milestone  dates  organized  in  a 
manner  to  achieve  optimal  progress  within  the  project.  Due  to 
multiple  project  changes,  the  most  common  compliance  issue  lies  in 
adhering  to  the  PMS  and  critical  deadlines  (Fleming  &  Koppelman, 
2012).EVMS  Criterion  7:  “ Identify  physical  products,  milestones, 
technical  performance  goals,  or  other  indicators  that  will  be  used  to 
measure  progress  (p.  3  796)  ” 

Similar  to  Criterion  6,  Criterion  7  must  identify  tangible  products  and 
deliverables  based  on  the  WBS  in  order  to  measure  earned  value  (Fleming  &  Koppelman, 
2012).  This  criterion  requires  contractors  to  describe  what  products  will  be  developed  or 
delivered  by  a  certain  milestone  date.  However,  the  major  conflict  with  this  criterion 
requires  contractors  to  clearly  identify  the  completed  phase  of  the  physical  product  or 
deliverable  in  accordance  with  the  schedule  (Fleming  &  Koppelman,  2012). 


c.  EVMS  Criterion  8:  “ Establish  and  maintain  a  time-phased  budget 
baseline  at  the  control  account  level,  against  which  program 
performance  can  be  measured  (p.  3803-3810 )” 

“Initial  budgets  established  for  performance  are  based  on  either  internal 
management  goals  or  the  external  customer-negotiated  target  cost,  including  estimates  for 
authorized  but  unpriced  work.  Budget  for  far  term  efforts  may  be  held  in  higher-level 
accounts  until  an  appropriate  time  for  allocation  at  the  control-account  level.  On  government 
contracts,  if  an  over-target  baseline  is  used  for  performance  measurement  reporting  purposes; 
prior  notification  must  be  provided  to  the  customer.” 

Criterion  8  requires  a  complete  and  formal  PMB  in  order  to  measure  all 
components  of  the  project  (Fleming  &  Koppelman,  2012).  Budgets  are  developed  and  work 
is  scheduled  to  be  completed  within  the  scope  of  the  project.  Fleming  and  Koppelman  (2012) 
emphasized  that  the  PMB  is  critical  to  establishing  a  framework  in  which  the  earned  value  of 
a  project  or  program  can  be  measured  in  accordance  with  time,  schedule,  and  performance. 
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d.  EVMS  Criterion  9:  “ Establish  budgets  for  authorized  work  with 
identification  of  significant  cost  elements  (labor,  material,  etc.)  as 
needed  for  internal  management  and  for  control  of  subcontractors 
(p.  3841)” 

Criterion  9  is  critical  to  ensuring  that  the  right  budgets  are  in  place  for  the 
project.  All  costs — to  include  material,  labor,  subcontracts,  and  travel — must  be  included  in 
the  budgeting  phase  (Fleming  &  Koppelman,  2012).  Furthermore,  Fleming  and  Koppelman 
(2012)  stated  that  budgets  must  be  assigned  to  respective  project  sections  and  any  budget 
changes  will  only  pertain  to  authorized  work  as  defined  in  the  WBS. 

e.  EVMS  Criterion  10:  “ To  the  extent  that  it  is  practical  to  identify  the 
authorized  work  in  discrete  work  packages,  establish  budgets  for  this 
work  in  terms  of  dollars,  hours,  or  other  measurable  units.  Where 
the  entire  control  account  is  not  subdivided  into  work  packages, 
identify  the  far-term  effort  in  larger  planning  packages  for  budget 
and  scheduling  purposes  (p.  3848-3855)” 

Criterion  10  focuses  on  clearly  defining  the  work  packages  within  the  WBS 
with  a  clear  distinction  for  the  near  and  long  term  (Fleming  &  Koppelman,  2012). 
Additionally,  established  budgets  containing  numerical  measures  (e.g.,  hours,  dollars)  must 
link  to  the  actual  work  function  to  be  performed  (Fleming  &  Koppelman,  2012). 

f.  EVMS  Criterion  12:  “ Identify  and  control  level-of-effort  (LOE) 
activity  by  time-phased  budgets  established  for  this  purpose.  Only 
that  effort  that  is  unmeasurable,  or  for  which  measurement  is 
impractical,  may  be  classified  as  LOE  (p.  3870)” 

Criterion  12,  the  remaining  criterion  in  the  planning,  scheduling,  and 
budgeting  group,  emphasizes  the  avoidance  of  level  of  effort  (LOE)  activities  such  as  “the 
project  manager  and  staff,  a  field  support  engineer,  guard  services”  (Fleming  &  Koppelman, 
2012,  p.  70).  These  activities  are  measured  by  the  passage  of  time  rather  than  actual  tangible 
deliverables  and  are  unable  to  be  measured  in  earned  value.  Thus,  Fleming  and  Koppelman 
(2012)  stated  that  LOE  activities  must  be  immediately  identified,  have  strict  budget  controls, 
and  be  avoided  to  the  maximum  extent. 
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3.  Group  4 — Analysis 

The  fourth  category  of  Analysis  contains  Criteria  23,  26,  and  27  and  requires  routine 
submission  of  EVM  data  such  as  cost  and  schedule  variances  to  maintain  effective  project 
management  (Dibert  &  Velez,  2006).  Any  variances  from  the  PMB  or  PMS  must  be 
evaluated  and  a  corrective  course  of  action  must  be  in  place  to  minimize  any  negative  impact 
to  the  project  (Dibert  &  Velez,  2006).  Fleming  and  Koppelman  (2012)  defined  Criterion  23 
as  follows: 


a.  EVMS  Criterion  23:  “ Identify ,  at  least  monthly,  the  significant 
differences  between  both  planned  and  actual  schedule  performance 
and  planned  and  actual  cost  performance,  and  provide  the  reasons 
for  the  variances  in  the  detail  needed  by  program  management  (p. 
4050)” 

Criterion  23  recommends  that  variances  in  schedule  and  cost  be  reported  on  a 
monthly  basis.  Once  a  variance  threshold  is  exceeded,  project  managers  must  determine  the 
cause  for  a  change  in  the  performance  measurement  baseline  to  include  developing  corrective 
actions  immediately  (Fleming  &  Koppelman,  2012).  Furthermore,  this  criterion  applies  to 
outside  suppliers  or  subcontractors  of  the  current  project.  A  main  issue  with  this  criterion  is 
a  lack  of  communication  between  the  contractor  and  subcontractor  in  EVM  reporting 
standards  and  managing  variances. 

b.  EVMS  Criterion  26:  “ Implement  managerial  actions  taken  as  the 
result  of  earned  value  information  (p.  4087)” 

Fleming  and  Koppelman  (2012)  suggested  a  process  that  ensures  corrective 
courses  of  action  are  implemented  as  a  result  of  exceeding  any  variance  threshold.  Criterion 
26  states  that  projects  may  have  a  set  point  at  which  cost  is  over  or  under  and  schedule  is 
ahead  or  behind,  at  which  management  should  implement  corrective  actions  to  lessen  any 
negative  impact  to  the  project  (Fleming  &  Koppelman,  2012). 
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c.  EVMS  Criterion  27:  “ Develop  revised  estimates  of  cost  at  completion 
based  on  performance  to  date,  commitment  values  for  material,  and 
estimates  for  future  conditions.  Compare  this  information  with  the 
PMB  to  identify  variances  at  completion  important  to  company 
management  and  any  applicable  customer-reporting  requirements, 
including  statements  of  funding  requirements  (p.  4095)” 

Criterion  27  focuses  on  the  project’s  estimate  at  completion  (EAC),  which 
should  be  continuously  updated  and  compared  against  actual  work  completed  and  the  initial 
budget  baseline  (Fleming  &  Koppelman,  2012).  Fleming  and  Koppelman  (2012)  stated  that 
authorized  work  must  be  budgeted  and  that  often,  different  project  sections  fail  to  accurately 
estimate  the  final  project  costs,  thus  distorting  the  estimate  of  costs  at  project  completion. 
Most  importantly,  EACs  should  frequently  be  compared  with  the  PMB  in  order  to  mitigate 
cost  issues  as  soon  as  possible  (Dibert  &  Velez,  2006). 

4.  Group  5 — Revisions 

The  last  category,  Revisions,  contains  Criteria  28  and  32,  which  require 
approved  changes  to  the  project  in  a  timely  manner  to  allow  for  integration  (Fleming  & 
Koppelman,  2012).  Fleming  and  Koppelman  (2012)  defined  Criterion  28  as  follows: 

a.  EVMS  Criterion  28:  “ Incorporate  authorized  changes  in  a  timely 
manner,  recording  the  effects  of  such  changes  in  budgets  and 
schedules.  In  the  directed  effort  prior  to  negotiation  of  change,  base 
such  revisions  on  the  amount  estimated  and  budgeted  to  the  program 
organizations  (p.  4134)” 

Fleming  and  Koppelman  (2012)  stated  that  Criterion  28  solely  focuses  on 
integrating  changes  in  a  timely  manner.  Defining  what  is  timely  poses  a  major  issue  for  this 
criterion  due  to  the  type  of  changes  that  would  need  to  be  merged  into  the  project  baseline 
(Fleming  &  Koppelman,  2012).  Additionally,  Fleming  and  Koppelman  (2012)  emphasized 
the  importance  of  ensuring  that  all  work  changes  are  immediately  documented  and  that  work 
value  is  estimated  into  the  PMB. 

b.  EVMS  Criterion  32:  “ Document  changes  to  the  PMB  (p.  41 70)  ” 

Criterion  32,  the  final  criterion,  serves  as  an  accounting  measure  to  ensure  that 
all  changes  are  documented  in  sequence  against  the  approved  PMB  (Fleming  &  Koppelman, 
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2012).  Project  changes  are  inevitable,  thus  a  tracking  system  of  changes  is  imperative. 
Failure  to  document  changes  to  the  PMB  defeats  the  purpose  of  EVMS  and  EVM  data 
management  (Dibert  &  Velez,  2006). 

C.  SUMMARY 

This  chapter  described  each  of  the  13  high-risk  EVMS  guidelines  in  detail,  which 
offers  a  better  understanding  of  how  to  determine  root  causes  and  severity  of  EVMS 
significant  deficiencies.  The  next  chapter  covers  the  extent  of  quantitative  risk  analysis  used 
in  DoD  acquisition  programs  and  presents  proposed  methodologies  for  this  project. 
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III.  QUANTITATIVE  RISK  ANALYSIS  IN  DOD  ACQUISITION  PROGRAMS 


A.  PREFACE 

The  implementation  of  quantitative  risk  analysis  in  DoD  acquisition  programs  is  the 
first  step  in  creating  a  framework  and  methodology  of  concepts,  topics,  issues,  and  data 
pertaining  to  our  research  topic.  Key  concepts  that  drive  our  progress  towards  achieving  our 
research  objectives  consist  of  the  following: 

•  quantitative  risk  analysis, 

•  operational  VaR,  and 

•  pairwise  comparison. 

Exploring  these  three  primary  areas  provides  the  link  between  the  research  questions, 
analysis,  and  application  of  the  operational  VaR  model  to  EVMS. 

B.  QUANTITATIVE  RISK  ANALYSIS 

The  use  of  quantitative  cost  risk  analysis  tools  can  be  valuable  in  measuring 
numerical  risk  to  the  government  (Galway,  2004).  However,  quantitative  risk  analysis  is 
rarely  utilized  in  DoD  acquisition  programs  because  the  methods  are  not  easily 
comprehended  by  project  managers  or  integrated  into  programs  (Fast,  2012).  In  fact,  Fast 
(2012)  discovered  that  quantitative  assessment  is  only  mentioned  once  in  the  Risk 
Management  Guide  for  DoD  Acquisition.  The  current  DoD  Risk  Management  Guide  (Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L],  2006) 
focuses  on  the  general  risk  management  process  to  mitigate  risks,  demonstrating  a  more 
qualitative  assessment  of  risk  (Fast,  2012). 

The  EVMS  is  used  as  a  risk  management  tool  for  the  overall  program  life  cycle,  but 
the  DoD  Risk  Management  Guide  does  not  address  a  quantitative  assessment  of  the  EVMS 
itself.  Galway  (2004)  practically  linked  project  quantitative  risk  assessment  to  EVM  by 
focusing  on  cost,  schedule,  and  performance  risks. 

Fast  (2012)  recommended  that  the  current  DoD  Risk  Management  Guide  be  revised  to 
include  more  quantitative  risk  analysis  methods.  The  DoD  Risk  Management  Guide 
(USD[AT&L],  2006)  identifies  three  components  of  risk:  a  future  root  cause,  probability  of 
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root  cause  occurring,  and  the  consequence  of  the  root  causes  occurring.  Fast  (2012) 
suggested  that 

simply  multiplying  the  probability  of  a  risk  event  (expressed  as  a  decimal)  by 
a  monetized  severity  of  loss  would  yield  a  monetary  risk  number.  Adding  up 
all  of  those  monetary  risk  numbers  for  all  elements  of  an  acquisition  program 
would  provide  an  infonnative,  albeit  somewhat  simplified  quantitative  risk 
assessment,  (p.  5) 

Therefore,  identifying  the  cost  to  correct  and  process  a  CAR  Level  III  deficiency 
would  lead  to  obtaining  a  monetized  severity  of  loss  in  dollars,  which  we  discuss  in  later 
chapters.  The  probability  of  a  significant  EVMS  deficiency  occurring  in  a  DoD  acquisition 
project  is  multiplied  by  the  monetized  severity  of  loss  of  that  deficiency.  This  calculation 
can  then  provide  a  quantitative  risk  value  that  could  justify  payment  withholdings,  which  we 
discuss  in  further  detail  in  Chapter  5. 

The  following  section  focuses  on  the  operational  VaR  approach  that  can  be  applied  in 
computing  the  VaR  for  significant  deficiencies  within  a  program’s  EVMS. 

C.  OPERATIONAL  VALUE  AT  RISK 

The  VaR  method  originated  in  the  1960s  when  economist  Harry  Markowitz 
discovered  the  concept  of  measuring  risk  for  bank  portfolio  assets  (Damodaran,  2007).  In  the 
last  few  years,  financial  analysts  and  statisticians  have  brought  forward  various  risk 
measurement  techniques  for  investment  traders  due  to  an  increase  in  trading  activities  and 
uncertainty  in  the  financial  market  (Hendricks,  1996).  In  business,  VaR  is  defined  as 
measuring  “the  worst  expected  loss  that  an  institution  can  suffer  over  a  given  time  interval 
under  normal  market  conditions  at  a  given  confidence  level”  (Butler,  1999,  p.  5). 

The  VaR  method  can  create  value  in  the  acquisition  workforce  by  estimating  the 
government’s  VaR  as  a  result  of  a  contractor’s  performance.  Thus,  the  VaR  method  is 
proposed  as  the  quantitative  risk  model  for  use  by  government  contracting  officers  when 
justifying  the  withholding  of  contractor  payments  for  a  significant  deficiency  in  the 
contractor’s  EVMS  (Fast,  2012). 

Jorion  (2007),  a  well-known  author  on  how  financial  institutions  and  the  insurance 
industry  calculate  financial  risk  using  the  VaR  method,  described  operational  VaR  as  “the 
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risk  of  loss  resulting  from  inadequate  or  failed  processes,  people,  and  systems  or  from 
external  events”  (p.  495). 

Fast  (2012)  described  operational  risk  as  the  cause  of  corporate  scandals  such  as  the 
Enron  scandal  in  2001  and  the  bankruptcy  of  Lehman  Brothers  in  2008.  Both  cases  are  an 
example  of  poor  corporate  accountability  and  transparency  such  as  a  compliance  failure, 
which  can  be  attributed  to  operational  risk.  Likewise,  operational  risk  in  government 
contracting  can  be  associated  with  the  risk  of  loss  resulting  from  failed  processes  or  systems 
such  as  the  EVMS. 

Operational  risk  is  based  on  the  frequency  and  severity  of  the  expected  and 
unexpected  losses  in  which  two  types  of  loss  distributions  can  occur.  High  frequency/low 
severity  losses  typically  consist  of  small  accounting  errors  due  to  daily  transactions  or  a  lack 
of  internal  control  (Fast,  2012).  These  are  expected  losses.  Alternatively,  low 
frequency/high  severity  losses  are  considered  disastrous  loss  events  that  pose  the  highest  risk 
to  an  organization  and  could  result  in  bankruptcy  (Fast,  2012).  These  are  unexpected  losses. 

Jorion  (2007)  measured  the  operational  VaR  using  four  steps: 

1 .  define  risk  categories  (processes,  people,  systems,  or  external  events), 

2.  measure  risk  factors  (loss  frequency  defined  as  the  number  of  loss  events  over 
a  set  time  frame), 

3.  measure  exposure  (loss  severity  defined  as  the  monetary  size  of  the  loss  once 
it  occurs),  and 

4.  calculate  risk  through  operational  value  and  expected  loss  (p.  497). 

Expected  loss  (EL)  is  how  much  an  organization  can  expect  to  lose  on  average  in 
daily  activities.  In  terms  of  EVMS,  EL  is  the  amount  lost  as  a  result  of  required  surveillance 
and  follow-up  actions  caused  by  deficient  guidelines.  For  example,  EL  is  the  government 
administrative  costs  of  monitoring  progress  towards  correcting  a  deficient  contract  and  the 
time  lost  to  developing  and  implementing  a  corrective  action  plan.  EL  can  be  measured 
using  the  following  equation: 

EL  =  E(n)  x  E(x)  (1) 

where 
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E(n)  =  loss  frequency  measured  as  the  number  of  loss  events  over  a  time  frame 
expressed  as  a  decimal  and 

E(x)  =  loss  severity,  expressed  in  dollars,  measured  as  the  size  of  the  loss  once  it 

occurs. 

Both  E (n)  and  E(x)  are  two  loss  distributions  that  are  combined  using  the  open  form 
solutions  method  in  which  the  mean  of  E (n)  and  of  E(x)  are  multiplied  to  get  the  mean  of  the 
aggregate  loss  distribution  (Navarrete,  2006).  The  mean  of  the  aggregate  loss  distribution 
then  becomes  the  EL. 

EL  =Mean  of  E(n)  x  Mean  of  E(x)  (2) 

Taking  the  kth  percentile  of  the  loss  distribution  or  range  of  expected  losses  then 
becomes  the  operational  VaR  (Navarrete,  2006).  Operational  VaR  can  be  calculated  using 
the  following  equation: 

Operational  VaR  =  kth  Percentile  of  ELs  (3) 

where 

kth  percentile  —  confidence  level  and 

ELs  =  loss  distribution/range  of  expected  losses  (E(n)  x  E(x)). 

The  unexpected  loss  (UL)  is  the  difference  between  the  operational  VaR  and  the  EL. 
UL  can  bankrupt  an  institution  and  is  the  amount  that  an  organization  should  expect  to  insure 
for  severe  loss  events  (Navarrete,  2006).  For  EVMS,  UL  is  the  amount  lost  to  the 
government  caused  by  the  potential  severity  that  a  certain  EVM  deficiency  can  cause  to  the 
contract  performance.  UL  can  be  calculated  using  the  following  equation: 

UL  =  Operational  VaR  -  EL  (4) 

Similarly,  the  expected  and  unexpected  losses  combined  equal  the  operational  VaR. 
For  the  contracting  officer,  EL  can  provide  a  VaR  for  expected  costs  caused  by  a  significant 
deficiency  such  as  administrative  costs  to  monitor  deficiencies.  UL  can  provide  the  ACO 
with  the  additional  VaR  that  the  significant  deficiency  can  cause  in  the  most  severe 
circumstances,  such  as  contract  termination.  Thus,  the  operational  VaR  can  be  summarized 
as  follows: 
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Operational  VaR  =  UL  +  EL  (5) 

Overall,  Jorion’s  (2007)  concept  of  the  operational  VaR  can  be  used  to  determine  the 
potential  loss  of  a  risky  asset  or,  in  terms  of  the  government,  the  potential  loss  in  dollars  of  a 
major  defense  acquisition  program  (MDAP)  regarding  people,  processes,  systems,  or  events. 
A  lack  of  compliance  in  the  EVMS  can  be  attributed  to  the  operational  risk  of  the  EVM 
process;  calculating  the  operational  risk  of  an  MDAP  EVMS  provides  the  government  with  a 
quantitative  tool  to  measure  EVMS  risk.  The  frequency  of  deficient  EVM  guidelines  can  be 
collected  using  historical  DCMA  CAR  data.  Chapters  4  and  5  demonstrate  that  a  severity 
analysis  of  the  13  high-risk  guidelines  will  result  in  a  rank  order  of  severity  for  the  guidelines. 
Last,  the  operational  VaR  would  be  able  to  provide  contracting  officers  with  a  monetary  risk 
amount  that  a  deficient  EVMS  presents. 

D.  PAIRWISE  COMPARISON 

Pairwise  comparison  has  been  utilized  for  many  years  as  a  tool  in  rank  ordering 
solutions  for  a  given  set  of  criteria.  The  basic  principle  of  the  method  is  to  compare  two 
solutions  and  award  each  solution  with  one  point  based  on  meeting  the  established  criterion; 
the  solution  with  the  most  points  wins  (Martin,  2011).  Summers  (2009)  identified  three  steps 
in  utilizing  the  pairwise  comparison  tool: 

1.  Create  a  table  for  each  criterion  that  has  the  potential  solutions  listed  in  the 
first  column  and  the  first  row; 

2.  Evaluate  each  of  the  solutions  with  respect  to  each  criterion  to  determine  if  the 
row  solution  is  better  (+1  point),  equivalent  (0  point),  or  worse  (-1  point)  than 
the  column  solution;  and 

3.  Total  the  sum  of  all  the  solutions  for  each  of  the  criteria. 
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Cost 

Honda  Civic 

Hyundai  Elantra 

Toyota  Corolla 

Honda  Civic 

-1 

+1 

Hyundai  Elantra 

+  1 

+1 

Toyota  Corolla 

0 

-1 

MPG 

Honda  Civic 

Hyundai  Elantra 

Toyota  Corolla 

Honda  Civic 

0 

+  1 

Hyundai  Elantra 

0 

+  1 

Toyota  Corolla 

-1 

-1 

Power 

Honda  Civic 

Hyundai  Elantra 

Toyota  Corolla 

Honda  Civic 

0 

+  1 

Hyundai  Elantra 

0 

+  1 

Toyota  Corolla 

-1 

-1 

Criteria/Solutions 

Honda  Civic 

Hyundai  Elantra 

Toyota  Corolla 

Cost 

0 

+2 

-1 

MPG 

+  1 

+  1 

-2 

Power 

+  1 

+  1 

-2 

Total  Score 

+2 

+4 

-5 

Figure  1.  Vehicle  Pairwise  Comparison  Example 


Figure  1  demonstrates  that  the  Flyundai  Elantra,  with  +4  points,  meets  the  majority  of 
the  set  criteria.  The  drawback  in  using  pairwise  comparison  is  the  inability  to  determine 
whether  a  solution  has  a  higher  weight  over  another  (Summers,  2009).  For  our  project,  the 
pairwise  comparison  tool  provides  a  structured  and  logical  methodology  by  comparing  one 
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guideline  criterion  to  another  in  order  to  determine  the  rank  order  severity  of  the  13  high-risk 
guidelines. 


E.  SUMMARY 

This  chapter  commenced  with  Fast  (2012)  suggesting  that  the  operational  VaR  can  be 
used  to  calculate  risk  in  EVMSs.  ACOs  can  utilize  quantitative  models  in  order  to  calculate 
a  VaR  amount  that  could  justify  withholding  contractor  progress  payments.  The  operational 
VaR  takes  into  consideration  the  frequency  of  a  loss  event  occurring  and  the  severity  of  the 
loss  to  calculate  the  VaR  due  to  failures  from  people,  processes,  or  systems.  Implementing 
the  operational  VaR  as  a  quantitative  DoD  risk  model  can  provide  contracting  officers  with  a 
tool  for  payment  withholds.  Furthermore,  the  pairwise  comparison  provides  a  method  to 
rank  order  the  severity  of  the  high-risk  guidelines  that  assist  in  calculating  the  severity  of  loss. 

The  next  chapter  describes  the  approach  to  collecting  and  analyzing  our  data. 
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IV.  METHODS  OF  ANALYSIS 


A.  ELECTRONIC  CORRECTIVE  ACTION  REQUEST  DATABASE 

Collectable  data  on  EVM  deficiency  reports,  also  known  as  CAR  reports,  serve  as  a 
key  element  for  this  research  project.  The  DCMA  manages  the  electronic  Corrective  Action 
Request  (eCAR)  database  that  consists  of  CAR  reports  ranging  from  levels  I-IV.  If  a 
contractor  is  found  deficient  during  their  EVMS  surveillance,  DCMA  ACOs  must  issue  a 
CAR  to  the  contractor  and  input  the  CAR  report  into  the  database  for  documentation.  As  a 
result,  the  DCMA  has  collected  and  organized  deficiency  reports  (CAR  reports)  for  over  200 
contracts  from  2007-2012.  The  e-CAR  database  organizes  information  in  the  following 
categories:  date  given  to  contractor,  contracts  by  manufacturing  facility  also  identified  as 
commercial  and  government  entity  (CAGE)  codes,  root  cause  of  deficiencies,  CAR  level, 
and  the  deficient  EVM  guideline  per  contract.  Most  importantly,  the  database  provided  us 
with  the  frequency  of  deficient  guidelines  by  CAGE  code  occurring  within  the  past  five  years. 

CAGE  Code  XX 

Frequency  of  EVMS  Deficient  Guidelines 

3  -i— 


uu  u 

n  i - 1  i  i - 1  i  i - 1 - 1 - 1  r 

GLl  GL3  GL6  GL7  GL8  GL9  GL10  GL12  GL23  GL26  GL27 

EVMS  High  Risk  Guidelines 

GL28  GL32 

Figure  2.  Example:  CAGE  Code  XX  Frequency  of  EVMS  Deficient 

Guidelines 


For  confidential  reasons,  each  CAGE  code  was  recoded  to  keep  company  information 
anonymous.  Figure  2  depicts  an  example  of  deficiencies  that  CAGE  Code  XX  received 
within  the  past  five  years.  We  chose  to  determine  loss  frequency  by  CAGE  code  because 
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EVM  systems  are  certified  and  validated  by  the  DCMA  on  a  facility  basis.  The  frequency  of 
each  guideline,  shown  in  Figure  2,  was  divided  by  a  total  of  1 1  deficiencies  that  spanned  over 
five  years  for  Facility  XX.  Table  3  lists  the  calculated  loss  frequency  percentage  of  each 
guideline  expressed  as  a  decimal.  Depending  on  the  total  amount  of  deficiencies  received  at 
the  facility,  ACOs  will  obtain  different  loss  frequency  percentages  for  each  CAGE  code. 

Table  3.  Example:  CAGE  Code  XX  Loss  Frequency  Percentage 


EVMS 

Guideline 

CAGE  Code 
XX 

Frequency 

Loss 

Frequency 

% 

1 

0 

0.000 

3 

1 

0.091 

6 

1 

0.091 

7 

0 

0.000 

8 

1 

0.091 

9 

0 

0.000 

10 

2 

0.182 

12 

1 

0.091 

23 

1 

0.091 

26 

1 

0.091 

27 

1 

0.091 

28 

2 

0.182 

32 

0 

0.000 

In  the  event  that  Facility  XX  receives  a  first-time  deficiency  in  Guideline  1,  all  loss 
frequency  percentages,  shown  in  Table  3,  would  change  correspondingly  based  on  a  new 
total  of  12  deficiencies. 


B.  METHOD  FOR  DETERMINING  EVM  GUIDELINE  RANK 
ORDER 

We  evaluated  the  rank  order  of  severity  posed  by  the  13  high-risk  guidelines  in  three 
sequential  stages.  In  the  first  stage,  we  rank  ordered  the  four  EVM  groups  and  their 
respective  guidelines  from  most  severe  to  least  severe,  based  on  our  analysis  and  supporting 
research. 
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Table  4.  EVM  Groups  and  Associated  High-Risk  Guidelines 


EVM  Group 

EVM  Guidelines 

Organization 

1,3 

Planning,  Scheduling, 
&  Budgeting 

6,  7,  8,9,  10,  12 

Analysis 

23,  26,  27 

Revisions 

28,32 

Second,  we  conducted  a  pairwise  comparison  between  the  guidelines  within  each 
group,  resulting  in  a  score  of  1  if  the  given  guideline  was  more  severe  than  the  other  and  a 
score  of  0  for  the  less  severe  guideline.  Table  5  outlines  20  pairwise  guideline  comparisons, 
which  are  segmented  by  their  respective  EVM  group. 
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Table  5.  20  Pairwise  Comparison  Combinations  of  Guidelines 


In  the  third  stage,  we  obtained  a  total  score  for  each  guideline  from  the  pairwise 
comparison  method,  using  Figure  3  as  our  score  card.  A  point  of  1  or  0  goes  into  each  blank 
cell.  Upon  conclusion  of  the  comparisons,  points  are  totaled  at  the  bottom  row  for  the  overall 
pairwise  score. 


\\  f, 1  '  :  '  V7UM  V 
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Using  Table  5  as  a  guide,  we  rank  ordered  each  guideline  within  its  respective  group 
based  on  its  pairwise  score.  Next,  we  combined  the  ranking  of  the  EVM  groups  and  the 
guidelines  within  each  group  to  configure  the  overall  ranking  from  1  through  13,  with  1 
ranked  as  the  most  severe  EVM  guideline.  For  example,  if  the  EVM  group  Organization  is 
ranked  as  the  most  severe  group  and  among  its  respective  guidelines  the  pairwise  comparison 
demonstrates  that  Guideline  3  is  more  severe  than  Guideline  1 ,  the  overall  ranking  results  in 
Guideline  3  as  the  most  severe  EVM  guideline  (rank  1  of  13).  Similarly,  if  the  EVM  group 
Revisions  is  ranked  as  the  least  severe  group  of  the  four  and  among  its  respective  guidelines 
the  pairwise  comparison  demonstrates  that  Guideline  28  is  less  severe  than  Guideline  32,  the 
overall  ranking  results  in  Guideline  28  as  the  least  severe  EVM  guideline  (rank  13  of  13). 

C.  METHOD  FOR  DETERMINING  THE  GUIDELINE  RISK 
FACTOR 

Upon  determining  the  overall  ranking  for  the  EVM  guidelines,  we  assigned  the  risk 
factor  for  each  guideline  using  the  rank  sum  method.  For  research  purposes  and  simplicity  of 
assigning  risk  factors,  we  chose  to  use  the  rank  sum  method  especially  so  we  could  develop  a 
rank  order  of  severity  for  each  of  the  13  high-risk  guidelines. 

However,  DCMA  officials  or  the  AGO  can  utilize  other  methods  to  best  fit  their 
specific  manufacturing  facility  or  contract  situation  in  determining  the  risk  factor.  For 
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example,  the  risk  factor  can  be  determined  using  the  direct  assessment  method  in  which  the 
DCMA  would  subjectively  assign  the  weighted  risk  for  each  deficient  guideline  depending 
on  its  assessment  of  guideline  severity  to  its  respective  manufacturing  facility.  For  example, 
the  first  six  guidelines  could  be  given  an  equal  weight  of  0.05  and  the  remaining  seven 
guidelines  could  be  given  an  equal  weight  of  0.10. 

For  this  research  project,  the  rank  sum  technique  takes  the  sum  of  the  ranks;  in  this 
case,  a  ranking  of  13  items  (1+2  +  3  +  4  +  5  +  6  +  7  +  8  +  9+10+11  +  12+  13)  provides  a 
total  sum  of  91.  To  obtain  the  risk  factor,  each  rank  item  is  then  divided  by  91.  The  highest 
factor  is  assigned  to  Rank  1,  and  each  consecutive  rank  item  is  assigned  to  the  next  highest 
factor  and  so  on.  Table  6  outlines  the  rank  sum  method  and  the  corresponding  risk  factor. 
The  risk  factor  can  be  a  useful  guide  for  ACOs  in  conducting  and  prioritizing  surveillance 
activities.  For  example,  higher  risk  factor  items  could  warrant  more  frequent  surveillance. 

Table  6.  Ranking  and  Risk  Factors 


Rank  Item 

Rank  Sum 
Calculation 

Risk 

Factor 

1 

13/91 

0.143 

2 

12/91 

0.132 

3 

11/91 

0.121 

4 

10/91 

0.110 

5 

9/91 

0.099 

6 

8/91 

0.088 

7 

7/91 

0.077 

8 

6/91 

0.066 

9 

5/91 

0.055 

10 

4/91 

0.044 

11 

3/91 

0.033 

12 

2/91 

0.022 

13 

1/91 

0.011 

Rank  Item  Sum:  91  Total: 

D.  SUMMARY 


The  eCAR  database,  consisting  of  EVMS  data  for  over  200  contracts  covering  the 
past  five  years,  is  an  essential  source  of  data  for  our  analysis  and  computation  of  the 
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operational  VaR.  The  eCAR  database  is  critical  to  obtaining  the  loss  frequency  by  CAGE 
code  of  the  high  risk  guidelines,  which  is  required  for  the  application  of  the  VaR  model. 
Establishing  the  rank  order  of  severity  for  the  EVM  guidelines  was  conducted  in  three  stages: 
rank  ordering  the  EVM  group,  utilizing  pairwise  comparison  to  rank  order  the  guidelines 
within  each  group,  and  then  combining  both  EVM  group  and  guideline  rankings  to  obtain  the 
overall  ranking  from  1  to  13.  The  risk  factors  for  each  rank  item  were  established  using  the 
rank  sum  methodology.  However,  DCMA  officials  have  the  discretion  to  use  other 
methodologies  that  best  meet  their  specific  facility  to  obtain  the  risk  factors,  which  can  be 
used  by  ACOs  to  prioritize  surveillance  activities. 

The  next  chapter  reveals  the  rank  order  of  severity  for  each  high-risk  guideline  and 
the  application  of  the  operational  VaR  to  EVMS. 
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V. 


ANALYSIS  AND  RESULTS 


A.  SEVERITY  ANALYSIS  OF  EVMS  GROUPS  &  GUIDELINES 

In  this  section,  we  discuss  our  findings  on  the  rank  order  of  severity  analysis  for  the 
13  EVMS  high-risk  guidelines.  The  rank  order  of  severity  can  assist  ACOs  in  prioritizing 
surveillance  activities  by  focusing  limited  resources  on  essential  high-risk  guidelines.  First, 
we  discuss  our  analysis  in  determining  the  EVMS  group  rank  order  of  severity.  Second,  we 
describe  the  reason  for  our  selection  in  each  of  the  20  pairwise  comparison  combinations  of 
the  high-risk  EVMS  guidelines. 

1.  EVMS  Groups 

As  described  in  the  introduction,  EVM  measures  a  program’s  cost,  schedule,  and 
performance  by  identifying  measures  to  ensure  that  a  program  is  in  accordance  with 
ANSI/EIA  748  standards  and  on  track  for  success.  Each  of  the  13  guidelines  belong  to  one 
of  the  four  EVMS  groups  covered  in  this  project.  Table  7  outlines  the  cluster  of  guidelines 
by  group,  which  were  ranked  from  Rank  1  (most  severe)  to  Rank  4  (least  severe). 
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Table  7.  EVMS  Group  Ranking 


Rank 

Order 

EVMS  Group 

EVMS 

Guidelines 

1 

Planning, 
Scheduling  & 
Budgeting 

6,  7,  8,9,  10,  12 

2 

Analysis 

23,  26,  27 

3 

Organization 

1,3 

4 

Revisions 

28,32 

The  following  demonstrates  our  reasoning  for  each  EVMS  group  ranking. 

•  Rank  1 :  Planning,  Scheduling  &  Budgeting 

We  ranked  this  group  as  most  severe  because  establishing  a  PMB  is  the  core  function 
of  this  group,  which  is  also  what  EVM  data  are  measured  against  to  monitor  program 
progress  and  risk  mitigation.  This  group  function  schedules  authorized  work  and  applies  the 
right  amount  of  resources  to  appropriately  budget  the  program;  these  measures  feed  into 
establishing  the  PMB  (Government  Accountability  Office  [GAO],  2009).  Common  root 
causes  of  poor  program  performance  include  issues  with  the  schedule  guidelines  within  this 
group,  such  as  too  many  activities  scheduled  with  unrealistic  time  frames.  Additionally,  the 
eCAR  database  demonstrates  that  the  sum  amount  of  frequencies  from  this  group  equate  to 
48.2%  of  the  total  frequencies  caused  by  the  13  high-risk  guidelines.  Failure  to  properly  plan, 
schedule,  authorize  work,  determine  a  budget,  and  develop  a  PMB  results  in  significant 
deficiencies  by  which  the  program  will  have  no  baseline  to  monitor  progress.  The  functions 
of  this  group  must  be  established  first  before  the  program  can  proceed  further. 


\i  ftAiiW*™  'cif.vnU)  y 
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•  Rank  2:  Analysis 

We  ranked  Analysis  as  the  second  most  severe  group  because  it  requires 
identification  and  analysis  of  cost  and  schedule  variances.  Analyzing  any  variance  within  the 
program  is  critical  to  developing  a  corrective  plan  of  action  to  mitigate  any  severe 
consequences.  This  functional  group  requires  analysis  of  the  reporting  measures  for  EVM 
such  as  cost  variances  (CV),  schedule  variances  (SV),  or  EACs,  which  are  critical  indicators 
of  program  performance.  Furthermore,  data  accuracy  is  a  common  indicator  of  poor  program 
performance.  Issues  such  as  lack  of  planning  for  corrective  actions,  multiple  data  input 
errors,  and  various  reasons  for  variances  play  a  role  in  ensuring  that  the  guidelines  within  this 
group  are  implemented  to  standard.  Additionally,  the  eCAR  database  demonstrates  that  the 
sum  amount  of  frequencies  from  this  group  equates  to  26.6%  of  the  total  frequencies  caused 
by  the  13  high-risk  guidelines.  This  group  requires  routine  evaluation  of  the  PMB  and  PMS 
as  a  safeguard  against  not  meeting  critical  deliverables  and  milestones.  Upon  establishing  a 
PMB  and  PMS,  routine  analytical  evaluation  is  essential  to  monitoring  EVMS  data  for 
accuracy  and  program  success. 

•  Rank  3:  Organization 

The  third  most  severe  group  is  Organization,  which  defines  the  scope  of  the  WBS  and 
organizational  responsibilities  for  each  WBS  package.  Upon  authorizing  the  WBS  to  include 
ensuring  accurate  data  analysis  of  the  authorized  work,  the  Organization  group  defines  the 
range  of  requirements  for  the  program  and  integrates  the  cost  and  scheduling  data  into  an 
organizational  structure.  Additionally,  the  eCAR  database  demonstrates  that  the  sum  amount 
of  frequencies  from  this  group  equates  to  13.9%  of  the  total  frequencies  caused  by  the  13 
high-risk  guidelines.  Organization  of  authorized  work  that  is  accurately  analyzed  is  essential 
prior  to  commencing  a  project. 

•  Rank  4:  Revisions 

We  ranked  the  Revision  group  as  least  severe  because  the  requirements  of  this  group 
are  implemented  after  a  PMB  is  established,  data  accuracy  is  evaluated,  and  organizations  of 
WBS  packages  are  conducted.  Revisions  are  a  mechanism  to  document  and  manage  changes 
in  a  timely  manner.  Additionally,  the  eCAR  database  demonstrates  that  the  sum  amount  of 
frequencies  from  this  group  equates  to  11.1%  of  the  total  frequencies  caused  by  the  13  high- 
risk  guidelines.  This  group  is  ranked  last  because  the  sequential  order  of  the  guidelines 
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within  this  group  can  only  be  conducted  after  the  requirements  in  the  other  three  groups  are 
executed. 

2.  EVMS  Guidelines 

The  pairwise  comparison  combinations  are  only  between  guidelines  (GLs)  within  the 
same  group.  Thus,  our  rank  order  analysis  is  limited  to  only  ranking  guidelines  based  on  the 
rank  of  their  respective  EVM  group.  For  example,  GL  1  could  be  ranked  within  the  top  five 
GLs  in  terms  of  most  severe;  however,  because  GL  1  belongs  to  the  Organization  group, 
which  is  ranked  3,  its  overall  rank  might  be  10  of  13. 

We  compared  each  pairwise  combination  based  on  selecting  the  guideline  that  posed 
a  higher  risk  defined  as  a  measure  of  future  uncertainties  in  achieving  program  performance 
goals  within  defined  cost  and  schedule  constraints.  We  evaluated  each  guideline  (significant 
deficiency)  in  terms  of  severity,  which  is  the  consequence  of  the  future  occurrence  of  that 
significant  deficiency.  The  following  sections  discuss  the  reasoning  for  each  selected  EVM 
GL  that  obtained  a  total  score  of  1  or  higher,  as  shown  in  Figure  4. 
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Figure  4.  Pairwise  Comparison  Scores 


a.  EVM  Guideline  1:  Define  Authorized  Work 

In  this  comparison,  GL  1  is  compared  with  GL  3,  as  shown  in  Table  8.  We 
selected  GL  1  as  more  severe  based  on  defining  the  range  of  requirements  for  work  using  the 
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WBS.  For  Tables  8  through  16,  the  highlighted  GL  is  more  severe  when  comparing  the  two 
GLs. 


Table  8.  GL  1  vs.  GL  3 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

1 

X 

GL  1  Define  Authorized  Work  (WBS  Elements) 

GL  3  Integrate  the  System 

The  WBS  is  an  essential  component  of  EVM  and  the  basis  to  correlate 
estimated  costs  and  schedule  with  actual  costs  (GAO,  2009).  A  program  must  have  an 
organized  WBS  structure  before  integration  of  the  system  can  take  place,  as  stated  in  GL  3. 
EVM  Criterion  3  integrates  all  system  inputs  such  as  costs,  schedules,  deliverables,  and  the 
WBS  structure,  to  include  identifying  team  roles  and  responsibilities.  For  example,  the  F-22 
Spiral  2  case  study  showed  that  the  integrated  master  schedule  (IMS)  and  cost  estimates  were 
tied  to  the  Spiral  2  WBS,  because  without  the  WBS,  the  F-22  team  would  not  be  able  to 
integrate  the  critical  inputs  (Dibert  &  Velez,  2006).  Additionally,  the  WBS  is  essential  to 
developing  the  PMB,  which  EVM  data  for  the  program  are  measured  against.  Since  the 
PMB  serves  as  the  baseline  for  measuring  EVM,  deficiencies  in  this  GL  are  linked  with 
deficiencies  in  GLs  1  and  6.  In  order  to  develop  a  well-defined  PMB,  there  should  not  be  a 
deficiency  in  GL  1  (development  of  WBS)  or  GL  6  (scheduling  authorized  work)  because 
both  GLs  are  required  to  develop  the  PMB. 

b.  EVM  Guideline  6:  Schedule  the  Work 

Table  9  shows  the  pairwise  comparisons  in  which  GL  6  was  selected  as  more 
severe  than  GLs  7,  9,  10,  and  12. 
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Table  9.  GL  6  vs.  GLs  7,  9,  10,  and  12 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

2 

X 

GL  6  Schedule  the  Work 

GL  7  Identify  Products,  Milestones,  and  Indicators 

3 

X 

GL  6  Schedule  the  Work 

GL  9  Establish  Budgets  for  Work 

4 

X 

GL  6  Schedule  the  Work 

GL  10  Identify  Work  Packages 

5 

X 

GL  6  Schedule  the  Work 

GL  12  Identify  and  Control  Level  of  Effort 

EVM  Criterion  6  was  selected  as  more  severe  based  on  developing  a  PMS, 


which  serves  as  the  source  for  scheduling  authorized  work.  Failure  to  accurately  establish 
critical  milestone  dates  in  an  organized  manner  highly  impacts  the  scheduling  variance.  In 
the  F-22  case,  the  PM  established  a  detailed  PMS;  however,  constantly  changing 
requirements  made  adhering  to  the  schedule  difficult  (Dibert  &  Velez,  2006).  The  numerous 
changes  created  a  negative  domino  effect  for  the  integrated  product  team  (IPT)  responsible 
for  developing  the  work  package  activity  schedules  (Dibert  &  Velez,  2006).  Scheduling  the 
work  feeds  into  the  SV,  which  is  a  direct  EVM  measure  of  success  and  a  deciding  factor 
towards  maintaining  or  canceling  a  program.  The  authorized  work  must  be  scheduled  before 
the  budget  or  work  packages  can  be  established  to  include  FOE  activities  as  described  in  GEs 
9,  10,  and  12. 


c.  EVM  Guideline  7:  Identify  Products,  Milestones,  and  Indicators 

Table  10  shows  that  the  following  pairwise  comparisons  identified  GL  7  as 
more  severe  than  GLs  9,  10,  and  12.  GL  7  describes  the  tangible  products  to  be  developed  in 
accordance  with  the  established  schedule. 
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Table  10.  GL  7  vs.  GLs  9,  10,  and  12 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

6 

X 

GL  7  Identify  Products,  Milestones,  and  Indicators 

GL  9  Establish  Budgets  for  Work 

7 

X 

GL  7  Identify  Products,  Milestones,  and  Indicators 

GL  10  Identify  Work  Packages 

8 

X 

GL  7  Identify  Products,  Milestones,  and  Indicators 

GL  12  Identify  and  Control  Level  of  Effort 

We  found  GL  7  to  be  more  severe  than  its  opponents  due  to  the  difficulty  in 


measuring  the  products  in  terms  of  value.  Furthermore,  contractors  must  clearly  identify  the 
completed  phase  of  the  physical  product  in  accordance  with  the  schedule  (Fleming  & 
Koppelman,  2012).  GL  7  is  a  challenging  criterion  to  meet,  especially  in  the  field  of 
software  development,  because  both  the  government  and  contractor  must  agree  on  the  value 
that  each  product  possesses,  which  can  be  complex  to  measure.  In  the  F-22  case  study, 
Lockheed  Martin  experienced  difficulty  in  developing  new  software,  which  led  to  problems 
in  identifying  the  measure  of  success  for  this  portion  of  the  program.  Furthermore,  F-22 
issues  within  this  criterion  resulted  in  special  emphasis  from  the  government.  The 
requirements  established  in  GL  7  are  linked  to  GLs  6  and  12  in  that  identifying  the  products 
must  be  in  accordance  with  scheduling  milestones  and  LOE  activities  must  be  identified 
during  the  activities  of  this  phase  as  well. 

d.  EVM  Guideline  8:  Plan  the  Performance  Measurement  Baseline 

In  the  following  comparisons,  GL  8  obtained  a  total  score  of  5  for  being 
selected  as  more  severe  than  GLs  6,  7,  9,  10,  and  12,  as  shown  in  Table  11. 
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Table  11.  GL  8  vs.  GLs  6,  7,  9,  10,  and  12 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

9 

GL  6  Schedule  the  Work 

X 

GL  8  Plan  the  Performance  Measurement  Baseline 

10 

GL  7  Identify  Products,  Milestones,  and  Indicators 

X 

GL  8  Plan  the  Performance  Measurement  Baseline 

11 

X 

GL  8  Plan  the  Performance  Measurement  Baseline 

GL  9  Establish  Budgets  for  Work 

12 

X 

GL  8  Plan  the  Performance  Measurement  Baseline 

GL  10  Identify  Work  Packages 

13 

X 

GL  8  Plan  the  Performance  Measurement  Baseline 

GL  12  Identify  and  Control  Level  of  Effort 

EVM  GL  8  serves  as  the  most  severe  criterion  based  on  the  critical  function  of 
the  PMB.  The  PMB  establishes  a  framework  to  measure  and  monitor  EVM  of  cost, 
schedule,  and  performance  of  a  program  (Fleming  &  Koppelman,  2012).  The  PMB  develops 
the  budget  and  funds  the  scheduled  work  authorized  for  the  program.  PMB  integrates 
schedule  and  cost  into  one  baseline.  Measurements  outside  of  the  baseline  signal  focus  areas 
for  the  program  management  team.  A  common  indicator  of  poor  performance  is  in 
developing  PMBs  in  which  budgets  are  unequally  distributed  to  earlier  scheduled  tasks,  thus 
concealing  issues  until  it  is  too  late  to  fix  them  (GAO,  2009).  DCMA  EVM  specialists  have 
stated  that  contractors  tend  to  only  create  a  PMB  constructed  on  the  base  year  rather  than  on 
the  life  cycle  of  the  program  due  to  the  current  budget  appropriations.  This  major  issue 
likely  leads  to  cost  overruns  and  can  severely  affect  the  credibility  of  program  officials  and 
put  the  program  at  high  risk  for  cancellation  (GAO,  2009).  Furthermore,  without  a  well- 
developed  PMB,  there  is  no  foundation  or  reference  point  to  measure  EVM  data  against 
performance. 


e.  EVM  Guideline  9:  Establish  Budgets  for  Work 

In  this  comparison,  GL  9  is  selected  as  more  severe  than  GL  12  because  it 
involves  establishing  budgets  for  the  authorized  work. 
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Table  12.  GL  9  vs.  GL  12 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

14 

X 

GL  9  Establish  Budgets  for  Work 

GL  12  Identify  and  Control  Level  of  Effort 

This  guideline  requires  an  appropriate  amount  of  funding  as  negotiated  for  the 


program;  however,  program  teams  tend  to  obligate  any  excess  funds  for  another  element  of 
the  program.  The  severity  involved  in  inappropriately  allocating  funds  elsewhere  leads  to  a 
deficiency  and  may  likely  affect  funding  in  later  phases  such  as  integration  and  testing 
(GAO,  2009).  Excess  funding  should  be  placed  in  management  reserve  (MR)  to  mitigate 
future  budgeting  risks.  Establishing  budgets  appropriately  from  the  beginning  is  crucial  to 
how  the  program  is  able  to  mitigate  financial  risks  throughout  the  life  cycle  of  the  program. 

f  EVM  Guideline  10:  Identify  Work  Packages 

Table  13  demonstrates  that  GL  10  was  selected  as  more  severe  in  comparison 
with  GLs  9  and  12  because  it  involves  identifying  work  packages  based  on  the  WBS  in  terms 
of  budgets,  hours,  and  other  EVM  numerical  measures  for  the  near  and  long  tenn  (Fleming  & 
Koppelman,  2012). 

Table  13.  GL  10  vs.  GLs  9  and  12 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

15 

GL  9  Establish  Budgets  for  Work 

X 

GL  10  Identify  Work  Packages 

16 

X 

GL  10  Identify  Work  Packages 

GL  12  Identify  and  Control  Level  of  Effort 

Before  a  budget  can  be  established,  the  work  package  measures  must  be 
identified  to  include  establishing  an  MR  for  future  uncertainties.  In  the  F-22  case,  work 


packages,  especially  outside  of  the  current  year,  were  difficult  to  identify  due  to  the 
constantly  changing  requirements.  Likewise,  because  Criterion  12  involves  identifying  LOE 
activities  that  measure  time  rather  than  tangible  products  that  can  be  measured  using  EVM,  it 
was  rated  as  the  least  severe  in  this  EVM  group.  LOE  activities  provide  no  value  to  EVM 
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because  they  do  not  contribute  to  measuring  SV  or  CV;  however,  LOE  activities  must  be 
identified  to  ensure  that  PMs  do  not  account  for  this  as  EVM  measurements. 

g.  EVM  Guideline  23:  Analyze  Schedule  and  Cost  Variances 

As  shown  in  Table  14,  GL  23  was  selected  as  more  severe  in  comparison  with 
GLs  26  and  27. 

Table  14.  GL  23  vs.  GLs  26  and  27 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

17 

X 

GL  23  Analyze  Schedule  and  Cost  Variances 

GL  26  Implement  Managerial  Actions 

18 

X 

GL  23  Analyze  Schedule  and  Cost  Variances 

GL  27  Develop  Revised  Estimates  of  Cost  at  Completion 

We  based  our  decision  on  the  importance  of  accurately  identifying  EVM  data 


such  as  variances  in  cost  and  schedule.  The  EVM  data  obtained  in  this  criterion  are  critical 
to  measuring  the  health  status  of  a  program  (GAO,  2009).  Furthermore,  by  requiring  a 
thorough  analysis  of  the  causes  for  exceeding  a  given  threshold,  management  can  better 
develop  a  course  of  action  to  fix  the  variances.  GL  23  provides  the  data  to  measure  program 
status,  for  without  it,  there  is  no  EVM  to  measure,  thus  defeating  the  purpose  and  benefits  of 
the  EVMS. 


h.  EVM  Guideline  26:  Implement  Managerial  Actions 

In  this  comparison,  Table  15  shows  that  GL  26  was  selected  as  more  severe 

than  GL  27. 


Table  15.  GL  26  vs.  GL  27 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

19 

X 

GL  26  Implement  Managerial  Actions 

GL  27  Develop  Revised  Estimates  of  Cost  at  Completion 

Although  common  indicators  of  poor  performance  show  frequent  deficiencies 
in  revising  and  updating  EACs  in  accordance  with  the  PMB,  as  described  in  GL  27, 
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implementing  managerial  actions,  as  described  in  GL  26,  affects  how  EACs  are  updated.  In 
the  F-22  case,  GL  26  was  rated  as  insufficient  for  failure  to  take  necessary  actions  to  realign 
the  program  based  on  the  variances,  which  led  to  inaccurate  EACs  (Dibert  &  Velez,  2006). 
EACs  must  be  compared  to  the  PMB  on  a  continual  basis,  but  if  the  necessary  managerial 
actions  are  not  taken  to  reconfigure  the  PMB  based  on  any  schedule  and  cost  variances, 
EACs  are  to  be  calculated  accurately.  GAO  (2009)  showed  that  many  programs  tend  to 
develop  overly  optimistic  EACs,  have  no  reasonable  plan  to  achieve  the  EAC,  and  fail  to 
account  for  risks.  The  eCAR  database  shows  that  GL  27  is  the  most  frequent  deficiency  at 
13.7%  for  all  13  EVM  guidelines.  Given  that  GL  27  is  highest  rate  of  deficiency  clearly 
shows  the  importance  of  ensuring  that  GLs  23  and  26  are  implemented  correctly  to  prevent 
deficiencies  in  GL  27.  Failure  to  implement  corrective  actions  results  in  poorly  defined 
EACs;  therefore,  GL  26  poses  a  higher  risk  in  severity. 

i.  EVM  Guideline  28:  Incorporate  Changes  into  Plans,  Budgets,  and 
Schedules 

In  our  last  comparison,  we  chose  GL  28  to  be  more  severe  in  consequence 
than  GL  32.  The  intent  of  GL  28  is  to  realistically  update  the  PMB  so  that  frequent  changes 
to  a  program  are  prevented.  Incorporating  the  changes  in  a  timely  manner  ensures  that  the 
execution  of  changes  is  implemented.  On  the  other  hand,  documenting  the  changes 
sequentially  through  tracking  a  record,  as  described  in  GL  32,  should  occur  once  the  changes 
are  incorporated.  We  found  GL  32  to  be  more  of  a  routine  administrative  function  that  bears 
less  risk  if  not  executed. 

Table  16.  GL  28  vs.  GL  32 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

20 

X 

GL  28  Incorporate  Changes  into  Plans,  Budgets,  and 

Schedules 

GL  32  Document  Changes  to  the  PMB 

B.  RANK  ORDER  RESULTS 

Upon  ranking  the  four  EVMS  groups  and  conducting  the  pairwise  comparison  of  the 
EVMS  guidelines  within  their  respective  groups,  we  used  the  pairwise  scores  to  determine 
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the  overall  ranking  for  all  13  EVMS  high-risk  guidelines,  ranking  each  guideline  from  1  as 
most  severe  to  13  as  least  severe.  Table  17  illustrates  the  results  of  the  guideline  pairwise 
score  and  rank  order. 

Table  17.  EVMS  Guideline  Pairwise  Score  and  Rank  Order 


EVM 

Rank 

Order 

Group  Ranking 

Guidelines 

Pairwise 

Score 

#  1 

#  1.  Planning, 
Scheduling,  & 
Budgeting 

8 

5 

#2 

6 

4 

#3 

7 

3 

#4 

10 

2 

#5 

9 

1 

#6 

12 

0 

#7 

#  2.  Analysis 

23 

2 

#8 

26 

1 

#9 

27 

0 

#  10 

#  3.  Organization 

1 

1 

#  11 

3 

0 

#  12 

#  4.  Revisions 

28 

1 

#  13 

32 

0 

Based  on  our  analysis  in  the  previous  section  and  the  implementation  of  our  rank 
order  methodology,  Table  18  lists  the  overall  rank  order  of  severity  for  the  13  EVMS  high- 
risk  guidelines. 
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Table  18.  Overall  Rank  Order  of  Severity 


It  a  nk 

Order 

EVMS 
High  Risk 
Guidelines 

#  1 

GL  8 

#  2 

GL  6 

#  3 

GL  7 

#  4 

GL  10 

#  5 

GL  9 

#  6 

GL  12 

#  7 

GL  23 

#  8 

GL  26 

#  9 

G I  27 

#  10 

G I  1 

#  11 

GL  3 

#  12 

GL  28 

#  13 

GL  32 

The  rank  order  of  severity  provides  a  means  for  the  AGO  and  his  or  her  staff  to  focus 
their  limited  resources  on  the  surveillance  of  high-risk  guidelines.  For  example,  Table  18 
suggests  that  GL  8  is  a  high-risk  priority  and  that  surveillance  should  be  conducted  weekly 
verses  quarterly  for  a  less  severe  guideline  such  as  GL  32. 

C.  APPLICATION  OF  THE  OPERATIONAL  VALUE  AT  RISK 
MODEL  TO  EARNED  VALUE  MANAGEMENT  SYSTEM 

As  discussed  in  Chapter  III,  the  operational  VaR  is  the  recommended  quantitative 
model  to  calculate  risk  for  a  given  contract  with  significant  EVMS  deficiencies.  This  section 
discusses  the  application  of  the  operational  VaR  model  for  two  different  deficient  EVMS 
scenarios.  For  both  examples,  we  chose  to  calculate  the  VaR  using  the  95%,  99%,  and  99.9% 
confidence  levels  based  on  historical  VaR  applications  demonstrating  that  financial  risk 
analysis  generally  ranges  from  95%  to  99.9%  (Navarrete,  2006).  However,  DCMA 
personnel  have  the  discretion  to  set  the  confidence  level  intuitively  based  on  the 
government’s  best  interest  and  specific  contract  situation. 

In  calculating  the  loss  frequency  of  each  deficient  guideline,  we  used  the  eCAR 
database  to  obtain  the  number  of  deficient  GLs  by  CAGE  code  per  year.  Given  the  data,  we 
developed  a  modified  method  to  obtain  the  loss  frequency  in  the  future.  Our  modified 
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method  to  obtain  the  loss  frequency  consisted  of  multiplying  the  time  factor  by  the  CAR 
weight: 

E (n)  =  Time  Factor  x  CAR  Weight  (6) 

where 

Time  Factor  =  (CARs  per  year  x  Time  Remaining  on  Contract)  (7) 

The  number  of  CARs  per  year  was  calculated  by  dividing  the  total  deficiencies  by  the 
number  of  year(s)  that  CARs  were  received.  The  CAR  weight  was  detennined  using  the 
eCAR  database  of  the  frequency  of  deficiencies  by  CAGE  code  per  year  and  obtaining  a 
probability. 

In  determining  the  loss  severity,  we  chose  to  use  both  the  DCMA  and  contractor’s 
administrative  and  labor  costs  involved  in  investigating  and  detennining  the  root  cause  of  a 
deficiency  and  processing  a  CAR.  The  costs  of  pursuing  a  CAR  involve  preparing  the 
corrective  action  plan,  implementing  the  corrective  action  plan,  independent  validation  that 
corrective  actions  were  actually  taken,  revalidation  of  the  EVMS  (if  necessary),  and  closing 
out  the  CAR.  Since  EVM  is  applied  to  all  incentive  and  cost  type  contracts  valued  at  $20 
million  or  above,  the  costs  to  run  the  EVMS  and  associated  costs  to  correct  significant 
EVMS  guideline  deficiencies  can  be  passed  by  the  contractor  to  the  government  as  allowable 
costs. 

In  addition,  any  cost  overruns  or  CV  that  a  specific  deficiency  caused  to  the 
contractor  and  government  would  also  be  taken  into  the  loss  severity  calculation.  The 
administrative  costs  of  fixing  a  CAR  and  cost  overruns,  if  applicable,  would  show  that  more 
costs  were  spent  for  work  accomplished  than  was  planned,  which  represents  risk  to  the 
government.  Because  we  did  not  have  data  on  CAR  severity  costs,  we  used  fictitious 
severity  data  for  both  EVMS  examples. 

The  operational  VaR  to  the  government  consists  of  both  EL  and  UL  at  a  given 
confidence  interval,  which  covers  the  government’s  expected  losses  and  severe  unexpected 
losses  from  a  deficient  guideline(s).  Figure  5  depicts  Navarrete’s  (2006)  illustration  of  how 
both  EL  and  UL  at  the  99.9%  confidence  level  make  up  the  operational  VaR  and  the  dollar 
amount  of  risk  that  the  government  suffers  as  a  result  of  the  deficiencies. 
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Each  example  follows  these  nine  steps: 

1.  Calculate  the  CAGE  code  CAR  weight  for  each  guideline.  Add  the  new 
guideline  deficiency  to  the  current  amount  of  guideline  deficiencies,  then 
divide  by  the  updated  total  amount  of  deficiencies  from  each  guideline. 

CAR  Weight i  = 

(Current  Guideline  Frequency  +  New  Guideline  Frequency) 

Total  Amount  of  Frequencies  From  Each  Guideline  (8) 

2.  Calculate  the  time  factor.  Determine  the  CARs  per  year  (total  amount  of 
CARs  divided  by  total  amount  of  years  that  CARs  were  received)  multiplied 
by  the  time  remaining  on  contract  (see  Equation  7). 

Time  Factor  =  (CARs  per  Year  x  Time  Remaining  on  Contract)  (9) 

3.  Calculate  E («),  the  loss  frequency,  by  multiplying  the  time  factor  by  each 
CAR  weight. 

E(«)i  =  Time  Factor  x  CAR  Weight  (10) 
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4.  Obtain  E(x)i,  the  loss  of  severity.  Use  the  administrative  and  labor  costs 
required  to  process  each  deficient  guideline  and  cost  overruns,  if  applicable, 
caused  by  the  guideline  deficiency. 

5.  Calculate  EL,  for  each  guideline  deficiency,  Multiply  Equation  10  by  E(x)i 
to  obtain  the  EL  value  for  each  guideline. 

EL,  =  E(n),  x  E(x),  (11) 

6.  Calculate  the  overall  EL.  Multiply  the  means  of  E (n)  and  E(x). 

Overall  EL  =  Mean  of  E(x)  x  Mean  of  E(n)  (12) 

7.  Calculate  the  operational  VaR.  Take  the  percentile  (established  confidence 
level)  of  the  loss  distribution  (range  of  ELs).  For  our  research,  we  calculated 
the  95th,  99th,  and  99.9th  percentile  using  the  Microsoft  Excel 
PERCENTILE. INC  function. 

Operational  VaR  =  Percentile  Range  of  EL(s)  (13) 

8.  Calculate  the  overall  UL.  Subtract  the  overall  EL  from  the  operational  VaR. 

Overall  UL=  Operational  VaR  -  Overall  EL  (14) 

9.  The  withhold  amount  caused  by  the  deficiencies  is  the  calculated  operational 
VaR  not  to  exceed  the  maximum  5%  of  contractor  progress  payments. 

1.  Scenario  1:  CAGE  Code  XX 

In  Scenario  1,  CAGE  Code  XX,  or  Contract  XX,  has  been  issued  a  CAR  Level  III 
with  the  following  repeated  EVMS  guideline  deficiencies:  3,  6,  10,  12,  23,  and  27.  Using 
unidentified  EVM  data  from  the  Defense  Cost  and  Resource  Center  (DCARC)  database, 
Table  19  outlines  pertinent  EVM  data  for  Scenario  1  with  guideline  severity  costs  and  any 
cost  overruns  as  a  result  of  the  CAR. 
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Table  19.  Scenario  1  Information 


Scenario  1:  CAGE  Code  XX 

Duration 

(Mo.) 

62 

Time  Remaining 
on  Contract 
(Mo.) 

24 

Budget  at 
Completion 
(BAC) 

$  729,000,000.00 

Total  Years  of 
CARs  Received 

5 

* 

Calculated 
Progress 
Payments  $ 
per  month 

$  11,758,064.52 

Max  Withhold 
Amount  $  per 
month 

$  587,903.23 

GL  Severity  Dollars 

EVMS 

Deficient 

Guidelines 

Administrative  $ 
Costs 

Overrun  $  Costs 

Total  $  Loss 
Severity  E(x) 

3 

$  75,000.00 

$  400,000.00 

$  475,000.00 

6 

$  110,000.00 

$  300,000.00 

$  410,000.00 

10 

$  50,000.00 

$  200,000.00 

$  250,000.00 

12 

$  20,000.00 

$  0 

$  20,000.00 

23 

$  40,000.00 

$  600,000.00 

$  640,000.00 

27 

$  95,000.00 

$  1,200,000.00 

$  1,295,000.00 

*Calculated  using  budget  at  completion  (BAC)  divided  by  duration 

Executing  the  nine  steps,  the  operational  VaR  for  the  six  EVMS  deficiencies  is 
calculated  as  follows: 

1.  Calculate  the  CAGE  code  CAR  weight  for  each  guideline.  Add  the  new 
guideline  deficiency  to  the  current  amount  of  guideline  deficiencies,  then 
divide  by  the  updated  total  amount  of  deficiencies  from  each  guideline. 

Table  20  illustrates  that  Guidelines  3,  6,  10,  12,  23,  and  27  are  repeated  deficiencies, 
which  caused  their  respective  CAR  weight  percentages  to  change. 
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Table  20.  Scenario  1:  Step  1  Calculations 


EVMS 

Guideline 

"Current" 
Cage  Code 
XX 

Frequency 

CAR 

Weight 

"New" 
Cage 
Code  XX 
Frequency 

New  CAR 
Weight 

1 

0 

0.000 

0 

0.000 

3 

1 

0.091 

2 

0.118 

6 

1 

0.091 

2 

0.118 

7 

0 

0.000 

0 

0.000 

8 

1 

0.091 

1 

0.059 

9 

0 

0.000 

0 

0.000 

10 

2 

0.182 

3 

0.176 

12 

1 

0.091 

2 

0.118 

23 

1 

0.091 

2 

0.118 

26 

1 

0.091 

1 

0.059 

27 

1 

0.091 

2 

0.118 

28 

2 

0.182 

2 

0.118 

32 

0 

0.000 

0 

0.000 

Total 

Deficiencies 

11 

New  Total 
Deficiencies 

17 

2.  Calculate  the  time  factor.  Determine  the  CARs  per  year  (total  amount  of 
CARs  divided  by  total  amount  of  years  that  CARs  were  received)  multiplied 
by  the  time  remaining  on  the  contract  (see  Equation  7). 

Table  21.  Scenario  1:  Step  2  Calculations 


CARs  per  year  =  (Total  Amount  of  CARS)  / 
(Total  Years  of  CARs  Received) 

Total 
Amount 
of  CARS 
Received 

Total 
Years  of 
CARs 
Received 

CARs  per 
year 

Time 

Remaining 

on 

Contract 

(yrs) 

17 

5 

3.4 

2 

Time  Factor  (CARs  per  year  X 
Time  Remaining  on  Contract) 

6.8 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-50- 


Calculate  E(n),  the  loss  frequency,  by  multiplying  the  time  factor  by  each 
CAR  weight.  CAR  weights  were  taken  from  the  calculations  in  Table  20. 

Table  22.  Scenario  1:  Step  3  Calculations 


E(n)  =  Time  Factor 
x  CAR  Weight 

EVMS 

Deficient 

Guidelines 

Time 

Factor 

CAR 

Weight 

Loss 

Frequency 

E(n) 

3 

6.8 

0.118 

0.80 

6 

6.8 

0.118 

0.80 

10 

6.8 

0.176 

1.20 

12 

6.8 

0.118 

0.80 

23 

6.8 

0.118 

0.80 

27 

6.8 

0.118 

0.80 

Obtain  E(x)i,  the  loss  of  severity.  Use  the  administrative  and  labor  costs 
required  to  process  each  deficient  guideline  and  cost  overruns,  if  applicable, 
caused  by  the  guideline  deficiency.  (See  Table  19.) 

Calculate  ELi  for  each  guideline  deficiency  by  multiplying  Equation  10  by 
E(x)i  to  obtain  the  EL  value  for  each  guideline.  Loss  severity  and  loss 
frequency  data  were  obtained  from  Tables  19  and  22,  respectively. 
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Table  23.  Scenario  1:  Step  4-6  Calculations 


EL  =  E(x)  x  E(n) 

EVMS 

Deficient 

Guidelines 

Loss  Severity  E(x) 

Loss 

Frequency 

E(n) 

Expected  Loss 
(EL) 

3 

$  475,000.00 

0.80 

$  381,140.00 

6 

$  410,000.00 

0.80 

$  328,984.00 

10 

$  250,000.00 

1.20 

$  299,200.00 

12 

$  20,000.00 

0.80 

$  16,048.00 

23 

$  640,000.00 

0.80 

$  513,536.00 

27 

$  1,295,000.00 

0.80 

$  1,039,108.00 

Mean  of  E(x) 

Mean  of  E(n) 

$  515,000.00 

0.87 

Mean  of  E(x)  x  Mean  of  E(n) 

Overall  EL  $448,050.00 

6.  Calculate  the  overall  EL.  Multiply  the  mean  of  E(n)  and  E(x).  (See  Table  23.) 

7.  Calculate  the  operational  VaR.  Take  the  percentile  (established  confidence 
level)  of  the  loss  distribution  (range  of  ELs).  For  our  research,  we  calculated 
the  95th,  99th,  and  99.9th  percentile  using  the  Microsoft  Excel 
PERCENTILE.INC  function.  (See  Figure  6  and  Table  24.) 


EVMS 

Deficient 

Guidelines 

Loss  Severity 
E(x) 

Loss  Frequency 
E(n) 

Expected  Loss 
(EL) 

3 

$  475,000.00 

0.80 

$  381,140.00 

6 

$  410,000.00 

0.80 

$  328,984.00 

Take  the  95th,  99th,  and  99.9th 

10 

$  250,000.00 

1.20 

$  299,200.00 

— 

Percentile  ofthe  Loss 

12 

$  20,000.00 

0.80 

$  16,048.00 

Distribution  (range  ofELs) 
using  the 

23 

$  640,000.00 

0.80 

$  513,536.00 

27 

$  1,295,000.00 

0.80 

$1,039,108.00 

J 

Function 

Mean  of  E(x) 

Mean  of  E(n) 

$ 

515,000.00 

0.87 

Mean  of  E(x)  x  Mean  of  E(n) 

Overall  EL 

$448,050.00 

Figure  6.  Scenario  1:  Step  7  Calculations 
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Table  24.  Scenario  1:  Step  7-8  Calculations 


Percentiles 

Losses 

Confidence 

Level 

Operational 

VaR  (Percentile 
of  Loss 
Distribution) 

Overall  EL 
(Aggregate 

Loss 

Distribution) 

Overall  UL 
(Operational 
VaR  -  EL) 

95.00% 

$  907,715.00 

$  448,050.00 

$  459,665.00 

99.00% 

$  1,012,829.40 

$  448,050.00 

$  564,779.40 

99.90% 

$  1,036,480.14 

$  448,050.00 

$  588,430.14 

8.  Calculate  the  overall  UL.  Subtract  the  overall  EL  from  the  operational  VaR. 
(See  Table  24  for  calculations.)  Figure  7  is  a  depiction  of  the  loss  distribution 
at  the  95th  percentile  that  shows  how  the  overall  EL  and  UL  equate  to  the 
operational  VaR. 


Loss  Distribution  @  95th  Percentile 


Loss  Severity  E(x) 


Figure  7.  Scenario  1:  Loss  Distribution  Chart  at  95th  Percentile 


9.  The  withhold  amount  caused  by  the  deficiencies  is  the  calculated  operational 
VaR  not  to  exceed  the  maximum  5%  of  contractor  progress  payments. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-53- 


Table  25. 


Scenario  1:  Total  VaR  and  Withhold  Amount  Detennination 


VaR  at  95% 
Confidence 
Level 

$  907,715.00 

Monthly 

Progress 

Payments 

$11,758,064.52 

Monthly 

Maximum 

Withhold 

Amount 

$  587,903.23 

Final 

Amount  to 
Withhold 

$  587,903.23 

Scenario  1  shows  that  the  total  operational  VaR  for  EVMS  guideline  deficiencies  (3, 
6,  10,  12,  23,  and  27)  is  approximately  $908,000.  The  monetary  risk  caused  by  the  EVMS 
deficiencies  exceeds  the  maximum  monthly  withhold  amount  of  approximately  $600,000. 
The  severity  of  the  six  EVMS  deficiencies  for  Contract  XX  were  equal  to  a  progress  payment 
withhold  of  7.7%,  which  exceeds  the  statutory  limit  of  withholding  a  maximum  5%  of 
progress  payments. 

2.  Scenario  2:  CAGE  Code  YY 

In  Scenario  2,  CAGE  Code  YY,  or  Contract  YY,  has  been  issued  a  CAR  Level  III 
with  never-before-seen  EVMS  guideline  deficiencies  1  and  32.  Using  unidentified  EVM 
data  from  the  DC  ARC  database,  Table  26  outlines  pertinent  EVM  data  for  Scenario  2  with 
guideline  severity  costs  and  any  cost  overruns  as  a  result  of  the  CAR. 
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Table  26.  Scenario  2  Information 


Scenario  2:  CAGE  Code  YY 

Duration 

(Mo.) 

41 

Time  Remaining 
on  Contract  (Mo.) 

24 

Budget  at 
Completion 
(BAC) 

$  198,000,000.00 

Total  Years  of 
CARs  Received 

2 

* 

Calculated 
Progress 
Payments  $ 
per  month 

$  4,829,268.29 

Max  Withhold 
Amount  $  per 
month 

$  241,463.41 

GL  Severity  Dollars 

EVMS 

Deficient 

Guidelines 

Administrative  $ 
Costs 

Overrun  $  Costs 

Total  $  Loss 
Severity  E(x) 

1 

$  60,000.00 

$  160,000.00 

$  220,000.00 

32 

$  15,000.00 

$ 

$  15,000.00 

*Calculated  using  BAC  divided  by  duration 

Executing  the  nine  steps,  the  operational  VaR  for  the  two  EVMS  deficiencies  is 
calculated  as  follows: 

1.  Calculate  the  CAGE  code  CAR  weight  for  each  guideline.  Add  the  new 
guideline  deficiency  to  the  current  amount  of  guideline  deficiencies,  then 
divide  by  the  updated  total  amount  of  deficiencies  from  each  guideline. 

Table  27  illustrates  that  Guidelines  1  and  32  are  never-before-seen  deficiencies, 
which  caused  their  respective  CAR  weight  percentages  to  change. 
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Table  27.  Scenario  2:  Step  1  Calculations 


EVMS 

Guideline 

"Current" 
Cage  Code 
XX 

Frequency 

CAR 

Weight 

"New" 
Cage 
Code  XX 
Frequency 

New  CAR 
Weight 

1 

0 

0.000 

1 

0.167 

3 

1 

0.250 

1 

0.167 

6 

0 

0.000 

0 

0.000 

7 

0 

0.000 

0 

0.000 

8 

0 

0.000 

0 

0.000 

9 

1 

0.250 

1 

0.167 

10 

0 

0.000 

0 

0.000 

12 

0 

0.000 

0 

0.000 

23 

0 

0.000 

0 

0.000 

26 

0 

0.000 

0 

0.000 

27 

1 

0.250 

1 

0.167 

28 

1 

0.250 

1 

0.167 

32 

0 

0.000 

1 

0.167 

Current 

Total 

Deficiencies 

4 

New  Total 
Deficiencies 

6 

2.  Calculate  the  time  factor.  Determine  the  CARs  per  year  (total  amount  of 
CARs  divided  by  total  amount  of  years  that  CARs  were  received)  multiplied 
by  the  time  remaining  on  the  contract  (see  Equation  7). 
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Table  28.  Scenario  2:  Step  2  Calculations 


CARs  per  year  =  (Total  Amount  of  CARS)  / 
(Total  Years  of  CARs  Received) 

Total 
Amount 
of  CARS 
Received 

Total 
Years  of 
CARs 
Received 

CARs  per 
year 

Time 

Remaining 

on 

Contract 

(yrs) 

6 

2 

3 

2 

Time  Factor  (CARs  per  year  X 
Time  Remaining  on  Contract) 

6 

Calculate  E(n),  the  loss  frequency.  Multiply  the  time  factor  by  each  CAR 
weight.  CAR  weights  were  taken  from  the  calculations  in  Table  27. 

Table  29.  Scenario  2:  Step  3  Calculations 


E(n)  =  Time  Factor 
x  CAR  Weight 

EVMS 

Deficient 

Guidelines 

Time 

Factor 

CAR 

Weight 

Loss 

Frequency 

E(n) 

1 

6 

0.167 

1.002 

32 

6 

0.167 

1.002 

Obtain  E(x)i,  the  loss  of  severity.  Else  the  administrative  and  labor  costs 
required  to  process  each  deficient  guideline  and  cost  overruns,  if  applicable, 
caused  by  the  guideline  deficiency.  (See  Table  26.) 

Calculate  ELi  for  each  guideline  deficiency.  Multiply  Equation  10  by  E(x)i 
to  obtain  the  EL  value  for  each  guideline.  Loss  severity  and  loss  frequency 
data  were  obtained  from  Tables  26  and  29,  respectively. 
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Table  30.  Scenario  2:  Step  4-6  Calculations 


EL  =  E(x)  x  E(n) 

EVMS 

Deficient 

Guidelines 

Loss  Severity  E(x) 

Loss 

Frequency 

E(n) 

Expected  Loss 
(EL) 

1 

$  220,000.00 

1.002 

$  220,440.00 

32 

$  15,000.00 

1.002 

$  15,030.00 

Mean  of  E(x) 

Mean  of  E(n) 

$  117,500.00 

1.002 

Mean  of  E(x)  x  Mean  of  E(n) 

Overall  EL 

$117,735.00 

6.  Calculate  the  overall  EL.  Multiply  the  means  of  E(n)  and  E(x).  (See  Table 
30.) 

7.  Calculate  the  operational  VaR.  Take  the  percentile  (established  confidence 
level)  of  the  range  of  EL(s).  For  our  research,  we  calculated  the  95th,  99th, 
and  99.9th  percentile  using  the  Microsoft  Excel  PERCENTILE. INC  function. 
(See  Figure  8  and  Table  31.) 


EVMS 

Deficient 

Guidelines 

Loss  Severity 
E(x) 

Loss  Lrequency 
E(n) 

Expected  Loss 
(EL) 

Take  the  95th,  99th,  and  99.9th 
Percentile  of  the  Loss 
Distribution  (range  ofELs) 
using  the 

Microsoft  Excel  Percentile 
Function 

1 

S  220,000.00 

1.002 

S  220,440.00 

32 

S  15,000.00 

1.002 

S  15,030.00 

Figure  8.  Scenario  2:  Step  7  Calculations 
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Table  31.  Scenario  2:  Step  7-8  Calculations 


Percentiles 

Losses 

Confidence 

Level 

Operational 

VaR  (Percentile 
of  Loss 
Distribution) 

Overall  EL 
(Aggregate 

Loss 

Distribution) 

Overall  UL 
(Operational 
VaR  -  EL) 

95.00% 

$  210,169.50 

$  117,735.00 

$  92,434.50 

99.00% 

$  218,385.90 

$  117,735.00 

$  100,650.90 

99.90% 

$  220,234.59 

$  117,735.00 

$  102,499.59 

8.  Calculate  the  overall  UL.  Subtract  the  overall  EL  from  the  operational  VaR. 
(See  Table  31  for  calculations.)  Figure  9  is  a  depiction  of  the  loss  distribution 
at  the  95th  percentile  that  shows  how  the  overall  EL  and  UL  equate  to  the 
operational  VaR. 


Loss  Distribution  @  95th  Percentile 

- w 

0.20 

"c 

LU 

EL  =  $117,735 

VaR  =  $210,170 

C 

m 

3 

u~ 

01 

GL  32 

(/) 

fig 

o 

— J 

0  05 

$3,006.00  $183,766.80 

Loss  Severity  E(x) 

Figure  9.  Scenario  2:  Loss  Distribution  Chart  at  95th  Percentile 


9.  The  withhold  amount  caused  by  the  deficiencies  is  the  calculated  operational 
VaR  not  to  exceed  the  maximum  5%  of  contractor  progress  payments. 
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Table  32. 


Scenario  2:  Total  VaR  and  Withhold  Amount  Detennination 


VaR  at  95% 
Confidence 
Level 

$  210,169.50 

Monthly 

Progress 

Payments 

$  4,829,268.29 

Monthly 

Maximum 

Withhold 

Amount 

$  241,463.41 

Final 

Amount  to 
Withhold 

$  210,169.50 

Scenario  2  shows  that  the  total  operational  VaR  for  the  EVMS  guideline  deficiencies 
(1  and  32)  is  approximately  $210,170.  In  this  case,  the  risk  caused  by  the  EVMS 
deficiencies  did  not  exceed  the  maximum  withhold  amount  of  $241,463.  The  severity  of  the 
two  EVMS  deficiencies  for  Contract  YY  equaled  to  a  progress  payment  withhold  of 
approximately  4.4%. 

D.  SUMMARY 

In  this  chapter,  we  have  analyzed  the  severity  for  each  of  the  13  high-risk  EVMS 
guidelines  and  developed  a  rank  order  of  severity.  By  obtaining  a  rank  order  of  severity,  we 
were  able  to  assign  each  guideline  with  a  weighted  risk  factor,  which  ACOs  can  use  in 
prioritizing  their  limited  resources  on  their  surveillance  of  high-risk  guidelines.  In  Scenario  1, 
the  operational  VaR  exceeded  the  federal  maximum  withhold  amount.  In  Scenario  2,  the 
operational  VaR  was  below  the  maximum  withhold  amount.  Both  scenarios  demonstrate  that 
ACOs  can  benefit  by  objectively  justifying  withhold  amounts. 

The  next  chapter  answers  each  research  objective,  provides  recommendations  for  the 
DCMA,  and  discusses  our  research  limitations. 
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VI.  SUMMARY,  CONCLUSION,  AND  FURTHER  RESEARCH 


A.  SUMMARY 

In  Chapter  1,  we  discussed  the  purpose  of  our  research  by  explaining  the  Pentagon’s 
new  business  system  rule  and  our  intent  to  focus  solely  on  EVMS.  The  new  business  system 
rule  addresses  deficient  contractor  business  systems  and  the  DCMA’s  authority  to  withhold 
5%  of  contractor  progress  payments  when  a  significant  deficiency  is  present.  Contractors  are 
held  more  accountable  and  punished  monetarily  for  failures  to  comply  with  EVMS 
guidelines  under  the  new  business  rule.  Our  research  efforts  benefit  the  DCMA  by 
recommending  a  quantitative  risk  analysis  tool  that  can  be  used  by  ACOs  to  justify  the 
amount  of  payment  withholds. 

In  Chapter  2,  we  provided  an  in-depth  overview  of  the  13  high-risk  EVMS  guidelines 
that  DCMA  EVM  specialists  asked  us  to  focus  on.  This  chapter  laid  the  foundation  to 
understand  root  causes  and  severity  of  EVMS  significant  deficiencies.  We  used  case 
examples  from  the  F-22  Spiral  2  program,  a  GAO  report  on  cost  estimation  and  assessment, 
and  EVMS  subject  matter  experts  in  providing  common  examples  of  deficiencies  for  each  of 
the  guidelines. 

In  Chapter  3,  we  described  quantitative  risk  analysis  in  DoD  acquisitions.  This 
chapter  provided  key  concepts  in  answering  our  research  objectives.  ACOs  can  calculate  the 
risk  to  the  government  by  applying  the  operational  VaR  model  to  justify  the  amount  to 
withhold  from  contractor  progress  payments.  The  operational  VaR  model  is  recommended 
because  it  takes  into  account  the  frequency  of  the  loss  event  occurring  and  the  severity  of  loss 
in  calculating  risk.  We  also  explored  the  pairwise  comparison  methodology  to  assist  us  in 
developing  a  rank  order  of  the  13  high-risk  EVMS  guidelines. 

In  Chapter  4,  we  discussed  the  eCAR  database  and  obtained  the  loss  frequency  of  the 
high-risk  guidelines,  which  is  required  for  the  application  of  the  operational  VaR  model. 
Establishing  the  rank  order  of  severity  for  the  EVM  guidelines  was  conducted  in  three  stages: 
rank  ordering  the  EVM  groups,  using  the  pairwise  comparison  score  to  rank  order  the 
corresponding  guidelines  within  each  group,  and  then  combining  both  EVM  groups  and 
guideline  rankings  to  obtain  the  overall  ranking  from  1  to  13.  The  risk  factors  for  each  rank 
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item  were  established  using  the  rank  sum  methodology,  which  provide  ACOs  with  a  means 
to  prioritize  their  surveillance  activities. 

In  the  last  chapter,  we  analyzed  the  risk  severity  that  each  guideline  posed  and 
developed  a  rank  order  of  severity  for  each  of  the  13  guidelines.  Upon  obtaining  a  rank  order 
for  each  guideline,  we  applied  the  operational  VaR  method  to  two  different  EVMS  guideline 
significant  deficiency  scenarios. 

B.  CONCLUSION 

The  intent  of  this  project  was  to  objectively  and  quantitatively  portray  EVMS  risk  in  a 
way  that  supports  a  monetary  withhold  decision  and  can  withstand  objection  (to  include 
litigation)  from  the  defense  contractor.  In  this  project,  we  hypothesize  that  using  quantitative 
risk  models  such  as  the  operational  VaR  method  and  simplification  of  that  business  model  for 
use  by  contracting  officers  provides  value  to  DCMA  professionals  in  the  implementation  of 
the  new  business  rule.  Our  project  focused  on  providing  the  DCMA  with  a  more  defensible 
risk  value  model  as  the  basis  for  withholding  contractor  payments.  In  order  to  assist  our 
research  efforts,  there  were  three  project  objectives.  The  following  are  the  findings  and 
recommendations  associated  with  each  research  objective: 

1.  Determine  whether  the  13  EVMS  high-risk  guidelines  can  be  grouped 
with  respect  to  root  causes  (causality  of  risk). 

a.  Findings 

This  research  objective  required  us  to  evaluate  the  rank  or  natural  order  to  the 
potential  severity  of  the  deficiency  posed  by  these  guidelines  and  to  assess  the  degree  of 
interdependence  or  causality  across  the  13  critical  EVMS  guidelines.  We  found  that  the 
high-risk  guidelines  were  already  assembled  within  an  EVM  group  for  which  the  group’s 
function  served  as  associated  root  causes  of  risk,  as  shown  in  Table  7. 

By  first  ranking  the  severity  of  the  EVM  group,  each  guideline  fell  into  a  category  of 
risk  for  a  program.  By  conducting  a  pairwise  comparison,  we  were  then  able  to  rank  order 
each  guideline  within  its  respective  EVM  group  to  determine  the  overall  rank  order  of 
severity,  as  shown  in  Table  17. 
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b. 


Recommendations 


Using  our  rank  order  methodology,  the  DCMA  could  conduct  a  more  formal 
method  of  rank  ordering  high-risk  guidelines  based  on  ACO  and  EVM  subject  matter  experts 
within  this  field.  An  accurate  rank  order  of  GL  severity  is  essential  to  prioritizing  the 
DCMA’s  limited  resources  to  fixing  and  monitoring  the  most  severe  GLs. 

2.  Evaluate  which  quantitative  method(s)  can  be  used  to  calculate  risk  value 
with  respect  to  noncompliance  with  both  critical  and  noncritical 
guidelines. 

a.  Findings 

In  this  research  objective,  we  found  the  operational  VaR  model  to  be 
applicable  in  calculating  risk  value  of  noncompliant  guidelines  and  identifying  quantitative 
definition  of  significant  deficiency.  In  Chapter  V,  there  are  two  scenarios  in  which  we 
calculate  the  monetary  risk  to  the  government  in  nine  steps.  By  obtaining  a  loss  frequency 
and  a  loss  severity  for  each  significant  deficiency,  we  were  able  to  calculate  the  total 
operational  VaR  for  EVMS  deficiencies.  We  found  that  the  monetary  amount  to  withhold 
differs  based  on  the  EVMS  significant  deficiencies  or  guidelines. 

b.  Recommendations 

The  nine  steps  recommended  for  calculating  the  operational  VaR  are  a 
valuable  quantitative  risk  analysis  tool  that  the  DCMA  could  implement  in  determining  or 
justifying  withhold  amounts  for  EVMS  deficiencies.  The  quantitative  model  is  objective  and 
removes  any  type  of  subjective  discretions  that  an  ACO  may  consider  in  the  amount  to 
withhold  from  the  contractor.  In  the  event  that  EVMS  deficiencies  calculate  to  more  than  5% 
of  progress  payments,  ACOs  will  have  the  confidence  to  withhold  the  maximum  allowed. 
Conversely,  if  EVMS  deficiencies  only  equate  to  3%  of  withholds,  the  contractor  will  feel 
confident  knowing  that  the  withhold  amount  was  not  based  on  the  discretion  of  the  ACO  but 
rather  an  established  quantitative  risk  analysis  tool. 
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3.  Determine  the  relationship  of  risk  value  calculations  and  findings  of 

EVMS  noncompliance  with  the  (a)  probability  of  error,  (b)  magnitude  of 

errors,  and  (c)  adverse  impact  of  errors 

a.  Findings 

By  gathering  data  from  the  eCAR  database,  we  were  able  to  obtain  the 
probability  of  error,  also  known  as  loss  frequency,  used  in  calculating  the  expected  loss.  By 
using  the  contractor’s  and  government’s  administrative  costs  of  pursuing  and  processing  a 
CAR  to  include  cost  overruns  as  a  result  of  the  deficiencies,  we  were  able  to  obtain  the 
magnitude  of  errors  (loss  severity).  The  mean  of  the  loss  severity  was  multiplied  by  the 
mean  of  the  loss  frequency  to  obtain  the  overall  expected  loss,  which  allowed  us  to  obtain  the 
operational  VaR  to  the  government,  also  known  as  the  adverse  impact  of  errors. 

We  found  that  by  using  the  operational  VaR  formula,  we  were  able  to 
calculate  EVMS  noncompliant  risks  by  obtaining  the  probability,  magnitude,  and  adverse 
impact  of  error  to  obtain  a  monetary  risk  value.  This  research  objective  required  us  to 
develop  a  deterministic  rule  set  that  yields  a  consistent  and  repeatable  finding  of  significant 
deficiency.  Thus,  we  found  the  nine  steps  to  calculate  the  operational  VaR  as  the  rule  set  that 
objectively  and  consistently  yields  the  risk  value  of  EVMS  significant  deficiencies. 

b.  Recommendations 

By  using  the  nine  steps  as  listed  in  calculating  the  operational  VaR  for  EVMS 
deficiencies,  ACOs  can  be  confident  in  withholding  calculated  monetary  amounts. 
Contractors  will  understand  the  importance  of  correcting  severe  deficiencies.  Severe 
deficiencies  hold  a  higher  monetary  risk  amount  to  the  government  and  vice  versa. 
Furthermore,  this  deterministic  rule  set  can  also  be  used  as  a  guide  for  corrective  action 
enforcement  by  putting  a  calculated  withhold  value  on  each  CAR  level  I  or  II  to  warn 
contractors  of  the  potential  payment  withholds  that  might  come  with  a  CAR  level  III  or  IV. 

C.  RESEARCH  SHORTCOMINGS 

Every  effort  has  been  made  to  gather  information  for  an  accurate  and  thorough 
severity  analysis  of  each  guideline.  Our  initial  methodology  included  conducting  a  pairwise 
comparison  of  the  high-risk  guidelines  through  a  survey  distributed  to  divisional 
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administrative  contracting  officers  (DACOs),  ACOs,  and  EVM  subject  matter  experts.  Due 
to  the  DCMA’s  legal  constraints,  the  survey  was  not  approved  in  a  timely  manner. 
Professional  input  from  DACOs,  ACOs,  and  DCMA  EVM  subject  matter  experts  would  have 
assisted  in  rank  ordering  the  EVM  groups  and  guidelines. 

We  recommend  that  in  the  future,  the  DCMA  conduct  a  formal  rank  order  analysis  of 
their  high-risk  guidelines  based  on  the  experience  and  working  knowledge  of  their  personnel 
in  order  to  develop  a  more  accurate  risk  factor  for  each  guideline.  The  survey  we  developed 
is  still  available  for  use  by  the  DCMA  (see  the  appendix). 

Secondly,  because  we  intended  to  distribute  a  survey,  we  decided  to  keep  the  number 
of  questions  limited  in  order  to  encourage  responses.  Thus,  we  only  conducted  a  pairwise 
comparison  among  guidelines  within  the  same  group  to  limit  the  questions  to  2 1 .  However,  a 
more  accurate  pairwise  comparison  should  have  compared  all  guidelines  to  one  another, 
which  would  have  resulted  in  78  comparisons  or  questions.  Furthermore,  our  pairwise 
methodology  limited  our  rank  order  analysis  to  ranking  guidelines  only  by  the  rank  of  their 
respective  EVM  group. 

Third,  we  recommend  that  the  eCAR  database  include  both  the  government’s  and 
contractor’s  administrative  and  labor  costs  associated  with  pursuing  and  processing  a  CAR  to 
completion  to  include  any  cost  overruns  that  the  deficiencies  cited  in  the  CAR  created  for  the 
government.  These  costs  are  considered  the  loss  severity  to  the  government  and  are  essential 
in  calculating  an  accurate  VaR  to  the  government. 

Last,  due  to  competition-sensitive  information,  we  were  unable  to  interview  the  ACO 
for  the  DCMA  at  Lockheed  Martin  (LM)  in  Sunnyvale,  CA.  This  DCMA  branch  was  one  of 
two  branches;  the  other  was  LM  in  Fort  Worth,  TX,  that  has  withheld  contractor  progress 
payments  since  the  implementation  of  the  new  business  rule.  Insight  into  the  actual 
circumstances  of  what  caused  the  ACO  to  withhold  payments  and  at  what  percentage  could 
have  highly  contributed  to  our  research  efforts. 

Despite  our  research  shortcomings,  the  information  gathered  and  analyzed  lays  out 
the  foundation  to  developing  a  more  accurate  rank  order  analysis  that  can  be  used  to 
realistically  calculate  the  operational  VaR  and  withhold  amount.  At  the  conclusion  of  this 
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project,  we  still  met  our  research  objectives  and  validated  that  the  operational  VaR  model  can 
be  used  as  a  defensible  risk  value  model  as  the  basis  for  withholding  contractor  payments. 

D.  AREAS  FOR  FURTHER  RESEARCH 

During  the  course  of  the  project,  we  identified  several  areas  for  further  research 
regarding  EVMS  deficiencies  and  the  VaR  model.  For  interested  researchers,  we 
recommend  the  following  for  future  areas  of  research: 

•  We  recommend  evaluating  the  root  causes  of  deficiencies  for  the  remaining 
19  guidelines  (2,  4,  5,  11,  13,  14,  15,  16,  17,  18,  19,  20,21,22,  24,  25,29,  20, 
and  31)  that  were  not  included  in  this  research.  We  also  recommend  ranking 
all  32  guidelines  for  severity  by  experienced  EVM  specialists. 

•  We  recommend  obtaining  contract  data  for  both  LM  Sunnyvale  and  LM  Fort 
Worth  and  applying  the  VaR  model  to  determine  if  there  were  any  differences 
in  a  quantitative  verses  a  subjective  withhold  amount  determination.  Once 
this  is  completed,  the  operational  VaR  model  should  be  verified,  validated, 
and  accredited  (VV&A)  by  an  independent  agency.  The  Director,  Cost 
Assessment  and  Program  Evaluation  (CAPE)  might  be  the  correct  level  of 
VV&A. 

•  Future  research  should  include  a  sensitivity  analysis  of  the  risk  factor 
determination,  pairwise  comparison,  and  VaR  model  to  recommend 
modifications  or  additions  to  the  model.  Sensitivity  analysis  would  examine 
the  sensitivity  of  inputs  to  the  VaR  model  based  on  changes  to  the  guidelines 
that  are  deficient. 

•  Other  areas  for  research  include  searching  for  best  practices  from  private 
industries  in  evaluating  risk.  Can  risk  management  models  from  other  private 
industries  be  applied  to  the  business  system  deficiencies?  The  variance- 
covariance  method,  Monte  Carlo  simulation,  risk  metrics,  and  other 
quantitative  approaches  can  be  researched  to  find  the  most  efficient  VaR 
model. 
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APPENDIX.  13  EYMS  HIGH-RISK  GUIDELINE  (GL)  SURVEY 


The  purpose  of  this  survey  is  to  evaluate  the  rank  or  natural  order  to  the  potential 
severity  of  the  deficiency  posed  by  the  selected  13  EVM  high  risk  guidelines 
(1,3,6,7,8,9,10,12,23,26,27,28,32).  This  research  project  excludes  the  accounting  EVM 
category  and  guidelines  and  focuses  on  only  13  guidelines  that  Senior  DCMA  Earned  Value 
Management  specialists  have  identified  as  high-risk  guidelines.  We  are  also  assessing  the 
degree  of  inter-dependence  or  causality  across  these  critical  EVMS  guidelines.  The  data  we 
collect  from  the  survey  will  assist  in  our  research  efforts  of  objectively  and  quantitatively 
portraying  EVMS  risk  in  a  way  that  supports  a  monetary  withhold  decision  from  the  defense 
contractor. 

You  are  invited  to  participate  in  a  research  study  titled  Deficient  Contractor  Business 
Systems:  Applying  the  Value  at  Risk  (VaR)  Model  to  Earned  Value  Management  Systems. 
We  would  like  you  to  participate  in  an  online  survey.  This  survey  will  consist  of  21 
questions.  During  the  survey  you  will  be  asked  to  provide  information  about  your 
experiences,  subject  matter  expertise,  and  professional  opinion  in  regards  to  the  EVMS 
process.  Participation  in  this  survey  is  only  voluntary  and  any  questions  you  do  not  answer 
will  be  respected. 

There  are  21  questions  in  this  survey  and  will  take  no  longer  than  20  minutes  to 
complete. 
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Rank  Order  of  the  EVM  Categories 


Based  on  your  experience,  rank  order  the  selected  EVM  Categories  (Organization,  Planning, 
Scheduling,  and  Budgeting,  Analysis,  or  Revisions)  from  most  severe  to  least  severe  in  deficiency. 

Each  category  should  be  evaluated  in  terms  of  the  severity,  which  is  the  consequence  of  the  future 
occurrence  of  a  significant  deficiency.  DFARS  Clause  252.234-7002  (a)  defines  a  significant 
deficiency  as  a:  shortcoming  in  the  system  that  materially  affects  the  ability  of  officials  of  the 
Department  of  Defense  to  rely  upon  information  produced  by  the  system  that  is  needed  for 
management  purposes. 

EVM  Categories 

Organization  (GL  1  and  3)  defines  the  range  of  requirements  prior  to  the  project  commencing. 
Planning,  Scheduling,  and  Budgeting  (GL  6,  7,  8,  9,  10,  and  12)  requires  a  management  control 
system  that  links  the  formal  planning,  scheduling,  and  budgeting  of  a  project  into  a  performance 
measurement  baseline  (PMB).  This  group  of  criterions  establishes  a  project  baseline  that  allows  for  a 
formal  means  of  project  discipline  and  assessment. 

Analysis  (GL  23,  26,  and  27)  requires  routine  submission  of  EVM  data  such  as  cost  and  schedule 
variances  to  maintain  effective  project  management.  Variances  should  be  evaluated  and  mitigated 
with  a  corrective  action  to  minimize  the  negative  impacts  on  the  project. 

Revisions  (GL  28  and  32)  require  approved  changes  to  the  project  in  a  timely  manner  to  allow  for 
integration. 

Click  on  an  item  in  the  list  on  the  left,  starting  with  your  highest  ranking  item,  moving  through  to  your  lowest 
ranking  item. 


Your  choices 


•  Group  1:  Organization 

•  Group  2:  Planning,  Scheduling,  and  Budgeting 

•  Group  4:  Analysis 

•  Group  5:  Revisions 
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Pairwise  Comparison  of  the  13  High  Risk  EVM  Guidelines 

For  each  of  the  20  questions  below,  select  the  guideline  (1, 3,  6,  7,  8,  9,  10,  12,  23,  26,  27, 
28,  or  32)  that  poses  a  higher  risk  defined  as  a  measure  of  future  uncertainties  in  achieving 
program  performance  goals  within  defined  cost  and  schedule  constraints. 

Each  guideline  (significant  deficiency)  should  be  evaluated  in  terms  of  the  severity,  which  is 
the  consequence  of  the  future  occurrence  of  that  significant  deficiency.  DFARS  Clause 
252.234-7002  (a)  defines  a  significant  deficiency  as  a:  shortcoming  in  the  system  that 
materially  affects  the  ability  of  officials  of  the  Department  of  Defense  to  rely  upon  information 
produced  by  the  system  that  is  needed  for  management  purposes. 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


•  ®  GL  1  Define  Authorized  Work  (WBS  Elements) 

•  ^  GL  3  Integrate  the  System 


Please  enter  your  comment  here:  — 


^^JChoosc  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


•  ^  GL  6  Schedule  the  Work 

•  ^  GL  7  Identify  Products,  Milestones,  and  Indicators 
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Please  enter  your  comment  here: 


^^JChoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 

•  ^  GL  6  Schedule  the  Work 

•  ^  GL  8  Plan  the  Performance  Measurement  Baseline  (PMB) 


Please  enter  your  comment  here: 


^^JChoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 

•  ^  GL  6  Schedule  the  Work 

•  ^  GL  9  Establish  Budgets  for  Work 


Please  enter  your  comment  here: 
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IChoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 

•  ^  GL  6  Schedule  the  Work 

•  ^  GL  10  Identify  Work  Packages 


Please  enter  your  comment  here: 


^JChoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


•  ^  GL  6  Schedule  the  Work 

•  ^  GL  12  Identify  and  Control  Level  of  Effort 


Please  enter  your  comment  here: 


^^JChoosc  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
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Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 

•  ^  GL  7  Identify  Products,  Milestones,  and  Indicators 

•  ^  GL  8  Plan  the  Performance  Measurement  Baseline  (PMB) 


Please  enter  your  comment  here: 


^^JChoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the  guideline 
to  be  a  higher  risk. 

Choose  one  of  the  following  answers 

•  ^  GL  7  Identify  Products,  Milestones,  and  Indicators 

•  ^  GL  9  Establish  Budgets  for  Work 


Please  enter  your  comment  here: 


^^JChoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the  guideline 
to  be  a  higher  risk. 

Choose  one  of  the  following  answers 

•  ^  GL  7  Identify  Products,  Milestones,  and  Indicators 
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GL  10  Identify  Work  Packages 


.  □ 


Please  enter  your  comment  here: 


^^JChoosc  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the  guideline 
to  be  a  higher  risk. 

Choose  one  of  the  following  answers 

•  ^  GL  7  Identify  Products,  Milestones,  and  Indicators 

•  ^  GL  12  Identify  and  Control  Level  of  Effort 


Please  enter  your  comment  here: 


^^JChoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the  guideline 
to  be  a  higher  risk. 

Choose  one  of  the  following  answers 

•  ^  GL  8  Plan  the  Performance  Measurement  Baseline  (PMB) 

•  ^  GL  9  Establish  Budgets  for  Work 
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Please  enter  your  comment  here: 


^^JChoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the  guideline 
to  be  a  higher  risk. 

Choose  one  of  the  following  answers 

•  ^  GL  8  Plan  the  Performance  Measurement  Baseline  (PMB) 

•  ^  GL  10  Identify  Work  Packages 


Please  enter  your  comment  here: 


^^JChoosc  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the  guideline 
to  be  a  higher  risk. 

Choose  one  of  the  following  answers 

•  ^  GL  8  Plan  the  Performance  Measurement  Baseline  (PMB) 

•  ^  GL  12  Identify  and  Control  Level  of  Effort 


Please  enter  your  comment  here: 
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Choose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 

•  ^  GL  9  Establish  Budgets  for  Work 

•  ^  GL  10  Identify  Work  Packages 


Please  enter  your  comment  here: 


^JChoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 

•  ^  GL  9  Establish  Budgets  for  Work 

•  ^  GL  12  Identify  and  Control  Level  of  Effort 


Please  enter  your  comment  here: 
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IChoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 

•  ^  GL  10  Identify  Work  Packages 

•  ^  GL  12  Identify  and  Control  Level  of  Effort 


Please  enter  your  comment  here: 


^^JChoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 

•  ^  GL  23  Analyze  Schedule  and  Cost  Variances 

•  ^  GL  26  Implement  Managerial  Actions 


Please  enter  your  comment  here: 


^^JChoosc  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
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Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 

•  ^  GL  23  Analyze  Schedule  and  Cost  Variances 

•  ^  GL  27  Develop  Revised  Estimates  of  Cost  at  Completion 


Please  enter  your  comment  here: 


^^JChoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 

•  ^  GL  26  Implement  Managerial  Actions 

•  ^  GL  27  Develop  Revised  Estimates  of  Cost  at  Completion 


Please  enter  your  comment  here: 


^^JChoosc  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 

•  ^  28  Incorporate  Changes  into  Plans,  Budgets,  and  Schedules 
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32  Document  Changes  to  the  PMB 


.  □ 


Please  enter  your  comment  here: 


JChoosc  only  1  guideline  from  the  following  pair 
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